[02:05:06] My pull request is ready for merging. [02:05:12] Username: thesenatorO5-q [02:05:17] -2 [09:17:11] (03CR) 10Giuseppe Lavagetto: [C: 03+1] Set testwiki API extractor to use the internal endpoint instead [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621206 (https://phabricator.wikimedia.org/T244843) (owner: 10Ladsgroup) [12:40:12] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] "Joe will deploy" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621206 (https://phabricator.wikimedia.org/T244843) (owner: 10Ladsgroup) [13:12:53] (03PS1) 10Ladsgroup: Fix the host [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621521 [13:13:39] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Fix the host [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621521 (owner: 10Ladsgroup) [13:26:57] (03PS1) 10Ladsgroup: Migrate rest of wikis to Envoy [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621522 (https://phabricator.wikimedia.org/T244843) [13:31:31] (03CR) 10Giuseppe Lavagetto: [C: 03+1] Migrate rest of wikis to Envoy [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621522 (https://phabricator.wikimedia.org/T244843) (owner: 10Ladsgroup) [13:34:16] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Migrate rest of wikis to Envoy [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621522 (https://phabricator.wikimedia.org/T244843) (owner: 10Ladsgroup) [13:46:20] (03PS1) 10Ladsgroup: Set the correct wikidata.org host [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621523 [13:46:59] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Set the correct wikidata.org host [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621523 (owner: 10Ladsgroup) [13:53:22] (03PS1) 10Ladsgroup: Several fixes [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621525 [13:56:15] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Several fixes [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/621525 (owner: 10Ladsgroup) [15:00:51] chrisalbon, kevinbazira, I just saw an icinga warning for ORES. Looks like it's a process count warning on ores1002 [15:01:06] Usually this happens when the machine runs out of memory and the OOM killer decides to crush celery. [15:01:19] I don't have time to look at it now, but I thought you should know. [15:01:46] killing celery increases the number of process? [15:02:01] ah, it probably is a "process count too low" [15:15:28] Ruh roh. [15:16:43] I _think_ it just fixed itself but Kevin and I can take a look [17:02:26] chrisalbon, I'm not sure, but I don't think the last deployment contained the huge memory savings that Pavol and haksoat1 were working on. [17:02:49] I think the last hangup was merging haksoat1's model building because one of the model builds used the wrong version of revscoring. [17:03:05] * halfak is stretching to remember last month's status. [17:06:57] I've got to run now, but I'm happy to help get those improvements into ORES when y'all are ready to start thinking about that. [17:07:08] It'll help with the level of alarm caused by these issues. [17:07:15] Take care [17:07:16] o/