[00:16:59] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:27:45] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [00:46:58] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:48] Project beta-scap-eqiad build #35910: FAILURE in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35910/ [01:05:35] Yippee, build fixed! [01:05:36] Project beta-scap-eqiad build #35911: FIXED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35911/ [03:46:57] Yippee, build fixed! [03:46:57] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #357: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/357/ [03:54:25] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #383: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/383/ [04:02:41] Yippee, build fixed! [04:02:41] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #444: FIXED in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/444/ [04:43:57] Yippee, build fixed! [04:43:57] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #411: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/411/ [04:54:49] Yippee, build fixed! [04:54:50] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #497: FIXED in 50 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/497/ [05:34:52] Project beta-scap-eqiad build #35938: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35938/ [05:44:56] Project beta-scap-eqiad build #35939: STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35939/ [05:55:15] Yippee, build fixed! [05:55:16] Project beta-scap-eqiad build #35940: FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35940/ [06:17:55] Yippee, build fixed! [06:17:55] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #372: FIXED in 3 min 2 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/372/ [06:26:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [06:36:31] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:39:30] qunit test for mediaviewer don't run anymore [06:39:55] see e.g. https://gerrit.wikimedia.org/r/#/c/182037/ [06:42:05] Yippee, build fixed! [06:42:05] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #218: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/218/ [06:46:23] last qunit job run I can find is from dec 8, first patch I can find where it should have run but didn't is dec 24 [06:46:51] this is for commits pending review; the qunit job still runs on merge [06:55:28] Project beta-scap-eqiad build #35946: FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35946/ [06:58:30] 3Continuous-Integration, MediaWiki-extensions-MultimediaViewer, Multimedia: MediaViewer Qunit job is not executed for new changesets - https://phabricator.wikimedia.org/T85434#946624 (10Tgr) 3NEW [07:00:23] 3Continuous-Integration, MediaWiki-extensions-MultimediaViewer, Multimedia: MediaViewer Qunit job is not executed for new changesets - https://phabricator.wikimedia.org/T85434#946630 (10Legoktm) qunit and phpunit tests don't run for non-whitelisted users. If you had +2'd the patch, they would run (and hopefully... [07:02:14] tgr: ^ [07:02:57] legoktm: thanks! is this a recent change? [07:03:05] no [07:03:31] been around since before I started contributing to MW. [07:06:10] Yippee, build fixed! [07:06:10] Project beta-scap-eqiad build #35947: FIXED in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35947/ [07:16:15] 3Continuous-Integration, MediaWiki-extensions-MultimediaViewer, Multimedia: MediaViewer Qunit job is not executed for new changesets - https://phabricator.wikimedia.org/T85434#946653 (10Tgr) 5Open>3Invalid a:3Tgr Wow, I was totally unaware. Is there a task about improving this behavior? I can see how runni... [07:36:55] tgr: there's a bug somewhere about running tests in isolated VMs and I think hashar had sent out emails about it at one point [07:57:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:22:06] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [08:41:38] 3Continuous-Integration: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#946702 (10Tgr) [09:55:42] 3operations, Beta-Cluster, Labs-Team: Backport new salt-syndic packages - https://phabricator.wikimedia.org/T85442#946773 (10yuvipanda) 3NEW [10:40:32] 3Quality-Assurance: Testing Pywikibot-Wikidata changes on non-production wikis - https://phabricator.wikimedia.org/T85358#946851 (10Qgil) [12:00:22] Project beta-scap-eqiad build #35975: FAILURE in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35975/ [12:01:59] 3Release-Engineering, operations: /usr/local/bin/deploy2graphite broken on tin due to nc command syntax - https://phabricator.wikimedia.org/T1387#946951 (10fgiunchedi) on which host does this happen? [12:05:51] Project beta-scap-eqiad build #35976: STILL FAILING in 4 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35976/ [12:10:09] Project beta-scap-eqiad build #35977: STILL FAILING in 3 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35977/ [12:39:39] Yippee, build fixed! [12:39:40] Project beta-scap-eqiad build #35978: FIXED in 25 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35978/ [12:58:41] (03PS1) 10XZise: Activate flake8 on Python 3 for pywikibot [integration/config] - 10https://gerrit.wikimedia.org/r/182067 [13:03:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:28:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:55:40] Project beta-scap-eqiad build #35987: FAILURE in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35987/ [14:05:27] Yippee, build fixed! [14:05:28] Project beta-scap-eqiad build #35988: FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35988/ [14:35:42] Project beta-scap-eqiad build #35991: FAILURE in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35991/ [14:45:05] Yippee, build fixed! [14:45:06] Project beta-scap-eqiad build #35992: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35992/ [14:53:36] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:27:48] 3Release-Engineering, operations: /usr/local/bin/deploy2graphite broken on tin due to nc command syntax - https://phabricator.wikimedia.org/T1387#947205 (10bd808) >>! In T1387#946951, @fgiunchedi wrote: > on which host does this happen? It happens on tin.eqiad.wmnet. [16:36:30] 3Release-Engineering, operations: /usr/local/bin/deploy2graphite broken on tin due to nc command syntax - https://phabricator.wikimedia.org/T1387#947220 (10fgiunchedi) ah I guessed right :) so nc itself is fine, however $MW_STATSD_HOST $MW_STATSD_PORT don't seem to be defined, unless they are set by the caller (... [16:55:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:57:58] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:05:13] Project beta-scap-eqiad build #36006: FAILURE in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36006/ [17:15:25] Yippee, build fixed! [17:15:25] Project beta-scap-eqiad build #36007: FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36007/ [17:20:05] (03CR) 10Legoktm: [C: 04-1] "I suppose this should be non-voting for now since it's currently failing." [integration/config] - 10https://gerrit.wikimedia.org/r/182067 (owner: 10XZise) [17:22:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:43] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:54:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [18:15:01] Project beta-scap-eqiad build #36013: FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36013/ [18:25:00] Project beta-scap-eqiad build #36014: STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36014/ [18:35:35] Yippee, build fixed! [18:35:35] Project beta-scap-eqiad build #36015: FIXED in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36015/ [18:35:37] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [18:38:42] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #166: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/166/ [18:50:30] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:03:15] Yippee, build fixed! [19:03:15] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #384: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/384/ [19:10:44] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #445: FAILURE in 52 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/445/ [19:19:41] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:20:27] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:35:35] Project beta-scap-eqiad build #36021: FAILURE in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36021/ [19:45:23] Yippee, build fixed! [19:45:24] Project beta-scap-eqiad build #36022: FIXED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36022/ [19:45:38] 3Beta-Cluster: m.wikidata.beta.wmflabs.org should point to a mobile IP - https://phabricator.wikimedia.org/T85469#947564 (10MaxSem) 3NEW a:3yuvipanda [19:49:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:49:45] 3Beta-Cluster: m.wikidata.beta.wmflabs.org should point to a mobile IP - https://phabricator.wikimedia.org/T85469#947576 (10yuvipanda) After digging around in the guts of OpenStackManager code, I discovered a simple 'Manage Domains' link in the sidebar for cloudadmins (incl. me). Have added this. [19:50:06] 3Beta-Cluster: m.wikidata.beta.wmflabs.org should point to a mobile IP - https://phabricator.wikimedia.org/T85469#947577 (10yuvipanda) Or rather, added the ability for @MaxSem to be able to point the DNS record to the mobile IP :) [20:00:02] 3Beta-Cluster: m.wikidata.beta.wmflabs.org should point to a mobile IP - https://phabricator.wikimedia.org/T85469#947588 (10yuvipanda) 5Open>3Resolved ``` YuviPanda, works now. just need to wait for dns cache to update and see if it works``` Yay :) [20:03:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [20:18:17] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #498: FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/498/ [20:40:44] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #423: FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/423/ [20:48:47] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:12:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:13:22] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #363: FAILURE in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/363/ [21:15:52] Heh. https://github.com/EFForg/https-everywhere/issues/460 [21:18:44] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:33:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:37:42] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:39] ryasmeen|Away: are you here today? [21:49:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:56:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:57:39] * chrismcmahon files some bugs without ryasmeen|Away [21:58:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:01:50] chrismcmahon: btw, we now have checks for the individual mediawiki servers [22:02:10] http://shinken.wmflabs.org/host/deployment-mediawiki01 for example [22:13:44] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #374: FAILURE in 17 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/374/ [22:19:19] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:23:38] Yippee, build fixed! [22:23:39] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #368: FIXED in 15 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/368/ [22:25:50] !log MaxSem created a DNS entry for m.wikidata.beta.wmflabs.org (copy from #-labs) [22:26:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [23:36:05] 3Continuous-Integration, Mobile-Web, MediaWiki-General-or-Unknown: QUnit job unable to collect logs, test still has active http requests writing to them - https://phabricator.wikimedia.org/T78590#948038 (10Jdlrobson) 5Open>3Resolved a:3Jdlrobson We haven't seen this for a while. This might be related to T8... [23:46:30] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%)