[07:09:15] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10elukey) [07:11:24] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10elukey) I have never deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/608812 but it may help in this case. Archiva runs on a ganeti VM, and current... [07:21:03] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Joe) >>! In T273003#6780172, @Dreamy_Jazz wrote: > Just to note that cyberbot I has been blocked because of the blanking issues on enwiki. The bot has also been blo... [07:26:30] 10Traffic: cp1087 needed a powercycle - https://phabricator.wikimedia.org/T273153 (10elukey) p:05Triage→03High [07:26:46] hello people, I just depooled and powercycled cp1087 --^ [08:02:55] uh [08:03:24] checking it ASAP [08:10:03] hmm purged seems to be kinda stressed out on cp1087 [08:10:11] it's consuming 4400% of the CPU :) [08:11:25] it was munching the backlog :) [08:11:36] https://grafana.wikimedia.org/d/RvscY1CZk/purged?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=cache_text&var-instance=cp1087&from=now-3h&to=now [08:19:46] considering that varnish-fe cache got wiped on the server restart it doesn't make a lot of sense IMHO [08:32:07] all looking good after a purged restart.. thanks for pinging elukey <3 [08:34:02] <3 [08:34:29] there were some memory errors, maybe we should replace the dimm [08:36:58] 10Traffic, 10SRE: cp1087 needed a powercycle - https://phabricator.wikimedia.org/T273153 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez Everything looking good.. purged had some troubles going through the backlog of PURGE requests... especially with varnish-fe. Considering that ats-be eventually caught... [08:37:56] elukey: I've seem some memory errors on eqsin being fixed with a FW upgrade.. I'll ping dcops [08:38:03] *seen [11:58:48] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10elukey) To keep archives happy: we deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/659236 to disable proxy buffering in nginx, pending verificatio... [13:13:52] 10Traffic, 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) > For a project having just few dependencies it takes 7 seconds from Maven Central compared to 25 seconds with Archiva. After the Nginx buffering has b... [17:06:27] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Cyberpower678) Got an influx of blank, 500s and 502 overnight. Your errors may be "few" overall, but they are happening in batches. Here's the log `lines=20 Dat... [17:07:32] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Cyberpower678) Looking at the log it looks like practically every request the bot made errored out. [17:11:19] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Cyberpower678) Timestamps of these entries seem to suggest that when the bot encounters these failures almost all API requests being made are erroring out during th... [17:17:11] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Dreamy_Jazz) >>! In T273003#6782955, @Joe wrote: >>>! In T273003#6780172, @Dreamy_Jazz wrote: >> Just to note that cyberbot I has been blocked because of the blanki... [17:19:41] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Cyberpower678) >>! In T273003#6784499, @Dreamy_Jazz wrote: >>>! In T273003#6782955, @Joe wrote: >>>>! In T273003#6780172, @Dreamy_Jazz wrote: >>> Just to note that... [18:06:16] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Urbanecm) 05Open→03Declined I'm taking the liberty to close this task as declined, for the following reasons: * The bot submits millions of requests per week,... [18:51:26] 10Traffic, 10SRE: Cyberbot is getting a lot of 502 errors, or blank responses when querying the API - https://phabricator.wikimedia.org/T273003 (10Legoktm) >>! In T273003#6784510, @Cyberpower678 wrote: > Actually the blanking is a consequence of these issues. Cyberbot queries index.php by using action=raw to... [23:18:47] 10Traffic, 10netops, 10Data-Services, 10cloud-services-team (Kanban): wikireplicas last-minute infra work to discuss / resolve - https://phabricator.wikimedia.org/T273248 (10BBlack)