[02:19:00] 10Traffic, 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, and 4 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10Shizhao) [02:22:58] 10Traffic, 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, and 4 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10cooltey) >>! In T262691#6456973, @Charlotte wrote: >>>! In T262691#6456908, @cooltey wrote: >> We have received a large number... [07:43:32] <_joe_> heads up: I'll be restarting lvs in eqiad and codfw to add a new LVS service [08:13:38] 10netops, 10Operations: every Sunday at 00:00 UTC, logrotate fails on netflow hosts - https://phabricator.wikimedia.org/T257128 (10ayounsi) 05Open→03Resolved a:03ayounsi T262751 had a more verbose error log: ` Sep 13 00:00:01 netflow1001 systemd[1]: Starting Rotate log files... Sep 13 00:00:01 netflow100... [09:25:12] 10netops, 10Operations: Upgrade Fastnetmon to 1.1.7 - https://phabricator.wikimedia.org/T257035 (10ayounsi) [09:55:41] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure: Beta cluster certificates have expired - https://phabricator.wikimedia.org/T262806 (10Mainframe98) [10:28:24] 10Traffic, 10Operations, 10Page Content Service, 10Platform Engineering, and 2 others: page/summary and page/mobile-html keeps responding code 429 in zh.wiki - https://phabricator.wikimedia.org/T262705 (10Jgiannelos) Hi @Pchelolo >>! In T262705#6454901, @Pchelolo wrote: > cc @Joe > > ratelimit was adde... [10:38:24] 10Wikimedia-Apache-configuration, 10Operations, 10Research, 10Patch-For-Review: Redirect wikimedia.org/research to research.wikimedia.org instead of some external closed survey - https://phabricator.wikimedia.org/T259979 (10Aklapper) After no feedback for one month, how to receive a review from #Operations... [13:22:55] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster certificates have expired - https://phabricator.wikimedia.org/T262806 (10Aklapper) Same as {T262816}? [13:24:29] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster certificates have expired - https://phabricator.wikimedia.org/T262806 (10Zoranzoki21) >>! In T262806#6458765, @Aklapper wrote: > Same as {T262816}? Looks so, I'm closing this one as duplicate. [13:24:50] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations: Beta cluster certificates have expired - https://phabricator.wikimedia.org/T262806 (10Zoranzoki21) [13:32:34] 10netops, 10Operations, 10ops-codfw: (Need by: ) codfw:rack/setup/new management switches - https://phabricator.wikimedia.org/T253154 (10Papaul) [13:36:37] 10Traffic, 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, and 5 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10Mholloway) I believe the search issue is separate from the zhwiki rate limiting. [15:53:30] 10Traffic, 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, and 4 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10ema) p:05Unbreak!→03High Users should not get 429s anymore, lowering priority to high while waiting for confirmation. For... [15:54:31] any traffickeers have a moment to give https://gerrit.wikimedia.org/r/627328 a stamp? [15:55:49] rzl: looking [15:58:57] rzl: you got stamps from two traffickers [15:59:01] ahaha [15:59:03] thanks both [16:05:23] disabling puppet on A:cp to merge this, will test on a canary host and then re-enable everywhere and let it roll out on the regular 30m schedule [16:13:47] done 👍 [16:18:32] nice [21:55:36] 10Traffic, 10netops, 10Operations: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10Dzahn) [21:56:46] 10Traffic, 10netops, 10Operations: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10Dzahn) [21:59:43] 10Traffic, 10netops, 10Operations: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10CDanis) An update on my last known disposition of the issue: * It appears to be an intermittent problem; individual outages are not long in duration, but there hav... [22:51:43] 10Traffic, 10netops, 10Operations: Wikimedia projects not reachable for some Telecom Italia users - https://phabricator.wikimedia.org/T262869 (10CDanis) Today we had reports of an issue from @Andyrom75 that was happening all the time on their Wind (AS1267) mobile connection, and was happening under some circ... [23:24:00] 10Traffic, 10Android-app-Bugs, 10Operations, 10Page Content Service, and 4 others: Incorrect language variant returned for PCS endpoints - https://phabricator.wikimedia.org/T249284 (10JoeWalsh) 05Open→03Resolved a:03JoeWalsh @holger.knust this bug is separate from T256491. It looks like the root caus...