[00:17:57] Project selenium-Flow » chrome,beta,Linux,BrowserTests build #446: 04FAILURE in 1 min 56 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/446/ [01:11:50] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10Patch-For-Review: Update to interwiki map - https://phabricator.wikimedia.org/T169979#3418319 (10Billinghurst) @Masti did it today, thanks for the reminder. [01:15:39] !log ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=enwiki (T170026, T165716) [01:15:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:15:44] T165716: No new data on ores_classification on beta labs since march memory issue - https://phabricator.wikimedia.org/T165716 [01:15:45] T170026: ORES spamming Beta Cluster's logstash - https://phabricator.wikimedia.org/T170026 [01:21:09] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 10ORES, 10Scoring-platform-team, 10User-Ladsgroup: ORES spamming Beta Cluster's logstash - https://phabricator.wikimedia.org/T170026#3417317 (10Ladsgroup) 05Open>03Resolved [06:21:10] Yippee, build fixed! [06:21:11] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #416: 09FIXED in 1 hr 41 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/416/ [06:38:09] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [06:48:08] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:15:30] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:15:32] Project beta-scap-eqiad build #163228: 04FAILURE in 1 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163228/ [08:26:06] Yippee, build fixed! [08:26:07] Project beta-scap-eqiad build #163229: 09FIXED in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/163229/ [08:55:33] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:36:20] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10Patch-For-Review: Update to interwiki map - https://phabricator.wikimedia.org/T169979#3418642 (10Dereckson) >>! In T169979#3418319, @Billinghurst wrote: > @Masti did it today, thanks for the reminder. Yes I've asked them to do on `#wikimedia-... [10:59:32] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:00:21] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<50.00%) [11:31:55] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:44:43] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:04:47] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:48:42] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:55:15] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:57:47] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:00:42] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:10:17] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:12:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:21] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:38] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:15:46] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:27:30] PROBLEM - Puppet errors on deployment-eventlogging03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:27:57] PROBLEM - Free space - all mounts on deployment-eventlogging03 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging03.diskspace.root.byte_percentfree (<50.00%) [19:31:56] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:23:24] PROBLEM - Puppet errors on castor is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]