[00:16:31] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1403.eqiad.wmnet'] ` and w... [00:16:59] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1402.eqiad.wmnet'] ` and w... [00:49:48] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1307.eqiad.wmnet'] ` and w... [07:23:56] !log restart uwsgi-ores on deployment-ores01 for T280420 [07:24:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:24:01] T280420: ores-beta.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T280420 [07:25:48] 10Beta-Cluster-Infrastructure, 10Machine-Learning-Team: ores-beta.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T280420 (10Majavah) I tried to restart it, without success: ` Apr 17 07:24:18 deployment-ores01 systemd[1]: uwsgi-ores.service: Failed with result 'timeout'. ` [10:07:05] 10LibUp: libup commit and upload upgrade twice for the same package on the same repo - https://phabricator.wikimedia.org/T280438 (10Umherirrender) [10:20:07] 10LibUp: LibUp: "None: None → None" in commit messages - https://phabricator.wikimedia.org/T277434 (10Umherirrender) Again - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/680607 [10:44:55] 10Beta-Cluster-Infrastructure, 10Machine-Learning-Team: ores-beta.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T280420 (10elukey) killed the uwsgi process and restarted, but it seems that any http query to port 8081 hangs. I see the following in the logs: ` 2021-04-17 07:24:20,082 WARNING or... [10:52:20] 10Beta-Cluster-Infrastructure, 10Machine-Learning-Team: ores-beta.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T280420 (10elukey) Celery fails with: ` Apr 13 17:31:19 deployment-ores01 systemd[1]: Started Celery workers. Apr 13 17:31:43 deployment-ores01 celery-ores-worker[23526]: Process Pr... [10:57:53] 10Beta-Cluster-Infrastructure, 10Machine-Learning-Team: ores-beta.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T280420 (10elukey) Also this is the status of uwsgi and celery after the restart: ` www-data 10152 1.9 11.3 1585260 923940 ? Ss 10:46 0:10 /srv/deployment/ores/deploy-cach... [11:30:40] 10Beta-Cluster-Infrastructure, 10Machine-Learning-Team: ores-beta.wmflabs.org is unreachable - https://phabricator.wikimedia.org/T280420 (10elukey) @Halfak can it be something related to your last change? Maybe it is missing something? [17:01:21] 10Beta-Cluster-Infrastructure: Beta cron jobs seem broken - https://phabricator.wikimedia.org/T277206 (10Aklapper) One month later, is this still an ongoing issue / existing problem? [17:02:51] 10Beta-Cluster-Infrastructure: Beta cron jobs seem broken - https://phabricator.wikimedia.org/T277206 (10Majavah) >>! In T277206#7011587, @Aklapper wrote: > One month later, is this still an ongoing issue / existing problem? Yes, this is still a problem that needs to be fixed. [17:39:27] 10phan-taint-check-plugin, 10Patch-For-Review: Array shapes not tracked when linking parameters to methods - https://phabricator.wikimedia.org/T269903 (10Daimona)