[00:56:15] 10Release-Engineering-Team (Kanban), 10MediaWiki-Cache, 10MediaWiki-Vagrant, 10Performance-Team, 10User-zeljkofilipin: MediaWiki core Selenium tests fail when targeting Vagrant - https://phabricator.wikimedia.org/T180035#3804964 (10Krinkle) [00:56:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Cache, 10MediaWiki-Vagrant, 10Performance-Team (Radar), 10User-zeljkofilipin: MediaWiki core Selenium tests fail when targeting Vagrant - https://phabricator.wikimedia.org/T180035#3744406 (10Krinkle) [01:43:00] PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [03:19:02] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [04:54:51] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:19:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:04:02] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:15:59] 10Release-Engineering-Team, 10Deployments, 10HHVM, 10Wikimedia-Incident: Figure out why HHVM kept running stale code for hours - https://phabricator.wikimedia.org/T181833#3803747 (10Legoktm) Are we sure this is an HHVM problem? It could have also been a problem with the InitialiseSettings.php cache not bei... [10:28:08] 10Continuous-Integration-Config, 10Pywikibot-core, 10Pywikibot-tests: pywikibot-core-tox-doc-docker timeouts sporadically - https://phabricator.wikimedia.org/T181881#3805276 (10Dalba) [10:31:28] PROBLEM - Free space - all mounts on integration-slave-docker-1002 is CRITICAL: CRITICAL: integration.integration-slave-docker-1002.diskspace.root.byte_percentfree (<44.44%) [11:28:09] 10Continuous-Integration-Config, 10Pywikibot-core, 10Pywikibot-tests: pywikibot-core-tox-doc-docker timeouts sporadically - https://phabricator.wikimedia.org/T181881#3805323 (10Dalba) This is probably related to the recent switch to docker in 9a0d299bb537. Increasing the timout might reolve the issue. [12:22:58] Yippee, build fixed! [12:22:59] Project selenium-GettingStarted ยป firefox,beta,Linux,BrowserTests build #604: 09FIXED in 57 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/604/ [13:33:04] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:41:27] PROBLEM - Free space - all mounts on integration-slave-docker-1002 is CRITICAL: CRITICAL: integration.integration-slave-docker-1002.diskspace.root.byte_percentfree (<11.11%) [14:08:01] RECOVERY - Puppet errors on deployment-trending01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:11:28] PROBLEM - Free space - all mounts on integration-slave-docker-1002 is CRITICAL: CRITICAL: integration.integration-slave-docker-1002.diskspace.root.byte_percentfree (<22.22%) [15:15:02] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:41] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:35:55] does Beta Cluster also use Meta's spam blacklist? [16:31:12] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 3 others: Plan migration of ORES repos to git-lfs - https://phabricator.wikimedia.org/T181678#3805805 (10Halfak) I've de-converted all of our github repos so that we can continue work while we wait for {T180628} [16:41:12] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 3 others: Plan migration of ORES repos to git-lfs - https://phabricator.wikimedia.org/T181678#3805812 (10Halfak) I've re-enabled observation on: https://phabricator.wikimedia.org/source/editquality https://pha... [17:02:38] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:26:17] Hey so bawolff and I found that when mwgate-php55lint is ran origin fetching times out see: https://integration.wikimedia.org/ci/job/mwgate-php55lint/13832/console any ideas why? [17:42:39] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:22] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:49:55] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<100.00%) [19:08:58] FYI found a bug with 1.30 - https://phabricator.wikimedia.org/T181898 [19:35:29] Reception123: its the weekend i wouldnt expect a response (or atleast a quick one during it) [19:36:13] Ok, though someone should definitely look before 1.30 is released [19:43:28] 10Release-Engineering-Team, 10User-Zppix: mwgate-php55lint fails to fetch origin - https://phabricator.wikimedia.org/T181899#3805997 (10Zppix) [19:44:04] ^ sorry if not right tags [19:50:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [20:40:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [23:25:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-Zppix: mwgate-php55lint fails to fetch origin - https://phabricator.wikimedia.org/T181899#3806260 (10Aklapper) -> Continuous-Integration tag [23:30:58] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:32:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-Zppix: mwgate-php55lint fails to fetch origin - https://phabricator.wikimedia.org/T181899#3806264 (10Paladox) I think this has happened before. I think @legoktm found it was the storage out of space. [23:35:02] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [23:36:09] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:25] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]