[00:56:17] PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 1043 MB (5% inode=91%); [02:26:20] PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 1006 MB (5% inode=91%); [02:26:39] 10Scoring-platform-team (Current), 10editquality-modeling, 10Patch-For-Review, 10Spanish-Sites, and 2 others: Train/test damaging & goodfaith models for Spanish Wikiquote (eswikiquote) - https://phabricator.wikimedia.org/T209784 (10Lsanabria) @Ladsgroup Sorry to comment on a closed task but the ORES filter... [03:56:23] PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 969 MB (5% inode=91%); [04:29:10] PROBLEM - puppet on ORES-worker01.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:57:10] RECOVERY - puppet on ORES-worker01.experimental is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [05:26:27] PROBLEM - check disk on ORES-web02.Experimental is WARNING: DISK WARNING - free space: / 932 MB (5% inode=91%); [06:26:12] RECOVERY - check disk on ORES-web02.Experimental is OK: DISK OK [06:27:45] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 0.018 second response time https://wikitech.wikimedia.org/wiki/ORES [06:46:55] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 540 bytes in 0.192 second response time https://wikitech.wikimedia.org/wiki/ORES [10:38:36] 10Scoring-platform-team, 10Cloud-VPS, 10artificial-intelligence, 10cloud-services-team (Kanban): GPU resources for Labs - https://phabricator.wikimedia.org/T159165 (10GTirloni) [10:40:57] 10Scoring-platform-team, 10Cloud-VPS, 10Documentation, 10cloud-services-team (Kanban): Document recommended process for installing vendor provided package upgrades in Wikimedia VPS - https://phabricator.wikimedia.org/T169247 (10GTirloni)