[01:14:25] 10Scoring-platform-team, 10CheckUser, 10Gamepress, 10Hashtags, and 9 others: pcaaaaaaaa - https://phabricator.wikimedia.org/T198094 (10Vvjjkkii) 05Invalid>03Open p:05Lowest>03High [01:19:42] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:24:41] 10Scoring-platform-team, 10CheckUser, 10Gamepress, 10Hashtags, and 8 others: xcaaaaaaaa - https://phabricator.wikimedia.org/T198086 (10Vvjjkkii) p:05Triage>03High [01:24:43] 10Scoring-platform-team, 10CheckUser, 10Gamepress, 10Hashtags, and 8 others: ycaaaaaaaa - https://phabricator.wikimedia.org/T198085 (10Vvjjkkii) p:05Triage>03High [01:47:42] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:48:35] PROBLEM - puppet on ORES-lb02.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:16:35] RECOVERY - puppet on ORES-lb02.Experimental is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [02:16:43] PROBLEM - puppet on ORES-worker02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:44:22] PROBLEM - puppet on ORES-worker05.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:49:42] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:12:21] RECOVERY - puppet on ORES-worker05.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:14:43] RECOVERY - puppet on ORES-worker02.experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:17:42] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:45:44] 10Scoring-platform-team, 10Multi-Content-Revisions, 10ORES: MCR support in ORES - https://phabricator.wikimedia.org/T195779 (10JJMC89) p:05High>03Triage [04:46:51] 10Scoring-platform-team, 10Wikilabels, 10artificial-intelligence: Implement a modeling self-check process - https://phabricator.wikimedia.org/T198144 (10JJMC89) p:05High>03Triage [05:09:34] 10Scoring-platform-team (Current), 10Analytics, 10CheckUser, 10EventBus, and 14 others: gzbaaaaaaa - https://phabricator.wikimedia.org/T195979 (10MusikAnimal) [05:13:13] 10Scoring-platform-team (Current), 10Analytics, 10CheckUser, 10EventBus, and 14 others: gzbaaaaaaa - https://phabricator.wikimedia.org/T195979 (10Community_Tech_bot) [06:21:55] 10Scoring-platform-team (Current), 10Analytics, 10EventBus, 10MediaWiki-JobQueue, and 4 others: ORESFetchScoreJob fails quite a lot - https://phabricator.wikimedia.org/T196076 (10Community_Tech_bot) [06:22:02] 10Scoring-platform-team (Current), 10Analytics, 10EventBus, 10MediaWiki-JobQueue, and 4 others: ORESFetchScoreJob fails quite a lot - https://phabricator.wikimedia.org/T196076 (10Community_Tech_bot) [07:27:43] 10Scoring-platform-team: Set up a docker-compose containing ORES services required to run tests - https://phabricator.wikimedia.org/T195077 (10Mainframe98) p:05High>03Triage [08:49:42] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 3 minutes ago with 18 failures. Failed resources (up to 3 shown): Package[ntp],Service[systemd-timesyncd],Service[exim4],Package[update-manager-core] [09:18:10] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:36:10] 10Scoring-platform-team (Current), 10ORES: ores1001 is in some kind of death spiral - https://phabricator.wikimedia.org/T194329 (10CommunityTechBot) [14:36:13] 10Scoring-platform-team (Current), 10ORES: ores1001 is in some kind of death spiral - https://phabricator.wikimedia.org/T194329 (10CommunityTechBot) [14:36:16] 10Scoring-platform-team (Current), 10ORES: ores1001 is in some kind of death spiral - https://phabricator.wikimedia.org/T194329 (10CommunityTechBot) p:05High>03Triage [14:36:18] 10Scoring-platform-team (Current), 10ORES: ores1001 is in some kind of death spiral - https://phabricator.wikimedia.org/T194329 (10CommunityTechBot) 05Open>03duplicate [14:36:20] 10Scoring-platform-team (Current), 10ORES: ores1001 is in some kind of death spiral - https://phabricator.wikimedia.org/T194329 (10CommunityTechBot) [14:42:43] 10Scoring-platform-team: deployment-ores01 needs more disk space - https://phabricator.wikimedia.org/T194315 (10CommunityTechBot) [15:12:12] 10Scoring-platform-team, 10Release-Engineering-Team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10CommunityTechBot) p:05High>03Triage [15:12:15] 10Scoring-platform-team, 10Release-Engineering-Team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10CommunityTechBot) [15:12:18] 10Scoring-platform-team, 10Release-Engineering-Team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10CommunityTechBot) [15:12:21] 10Scoring-platform-team, 10Release-Engineering-Team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10CommunityTechBot) [18:12:13] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES: Provide a user-accessible page showing current thresholds for each sensitivity level - https://phabricator.wikimedia.org/T195083 (10CommunityTechBot) [18:12:16] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES: Provide a user-accessible page showing current thresholds for each sensitivity level - https://phabricator.wikimedia.org/T195083 (10CommunityTechBot) p:05High>03Triage [18:12:19] 10Scoring-platform-team (Current), 10Global-Collaboration, 10MediaWiki-extensions-ORES: Provide a user-accessible page showing current thresholds for each sensitivity level - https://phabricator.wikimedia.org/T195083 (10CommunityTechBot)