[00:00:05] 10Continuous-Integration-Config, 10Release-Engineering-Team: jsduck custom tags error when trying to run docs for a skin - https://phabricator.wikimedia.org/T170703#3439898 (10Krinkle) This is not a CI problem. The draft commit at is creating a configur... [00:00:34] 10Continuous-Integration-Config, 10Release-Engineering-Team: jsduck custom tags error when trying to run docs for a skin - https://phabricator.wikimedia.org/T170703#3440854 (10Krinkle) 05Open>03Resolved a:03Krinkle Resolved in Code-Review. [00:13:21] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:21:43] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [00:24:00] 10Release-Engineering-Team (Kanban), 10Phabricator (2016-10-05), 10Regression, 10Upstream: Regression: Ability to comment on phame posts not working - https://phabricator.wikimedia.org/T144338#3440886 (10Paladox) 05Open>03Resolved Resolved again. Reported on #wikimedia-cloud but forgot that this task w... [00:25:51] 10Release-Engineering-Team (Kanban), 10Phabricator (2016-10-05), 10Regression, 10Upstream: Regression: Ability to comment on phame posts not working - https://phabricator.wikimedia.org/T144338#2596809 (10Zppix) Buttons are now grayed out for user w/o proper permissions. See attached screenshot: {F8762431} [00:48:23] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:14:24] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:24:04] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:42:01] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:49:04] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:54:23] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:03] RainbowSprinkles: how much do you (or someone else here) know about the Debian MediaWiki package? [02:14:31] I'm getting a very unusual fatal on wikiconference.org that may or may not be related to it: https://wikiconference.org/index.php?title=Special:CreateAccount [02:22:02] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [02:22:12] It was caused by an extension -_- [02:25:39] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:30:32] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:02:44] harej: I know zilch. Ask Kunal [03:05:34] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:14:48] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:15:40] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [03:32:44] harej: legoktm is the one to ask about the package [03:34:48] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [03:44:54] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:49:33] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:52:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [04:18:00] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #453: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/453/ [04:19:55] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:32:59] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [04:39:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:00:00] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:02:58] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [05:14:42] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:22:43] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [05:30:00] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [05:42:34] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:09:42] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:32:33] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:45] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [08:04:41] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [08:05:30] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:15:39] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:35:41] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [08:36:25] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:40:28] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:45:31] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:05:42] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [09:15:05] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:15:31] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:32:34] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:35:31] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:40:28] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [09:50:04] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:36:51] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:56:18] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:59:33] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:00:21] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [11:07:36] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:21:52] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [11:30:21] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:50:22] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:01:20] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:23:42] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:41:19] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [12:44:44] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:46:18] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:03:43] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [13:04:47] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:12:18] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:32:17] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:10] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #460: 04FAILURE in 2 min 9 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/460/ [14:04:05] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:05:11] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:24:05] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:43] Yippee, build fixed! [14:33:44] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #455: 09FIXED in 1 min 42 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/455/ [14:45:13] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:48:42] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:55:13] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:57:47] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:00:42] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:10:16] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:12:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:21] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:37] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:59] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:15:48] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:27:31] PROBLEM - Puppet errors on deployment-eventlogging03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:29:49] PROBLEM - Puppet errors on deployment-kafka03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:45:51] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:53:59] RECOVERY - Puppet errors on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [16:04:47] RECOVERY - Puppet errors on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:42] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:00:41] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:51] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:21:24] PROBLEM - Puppet staleness on deployment-kafka01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [17:27:57] PROBLEM - Free space - all mounts on deployment-eventlogging03 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging03.diskspace.root.byte_percentfree (<100.00%) [17:42:18] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:02:18] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:40:03] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:41:29] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:02:30] 10Continuous-Integration-Infrastructure: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#3441484 (10Krinkle) OK. `mw-error.log` and `mw-exception.log` are empty again for phpunit and qunit jobs. Let's stay on top of this and enfor... [19:03:25] 10Continuous-Integration-Infrastructure: Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown - https://phabricator.wikimedia.org/T50002#3441486 (10Krinkle) [19:09:26] Yippee, build fixed! [19:09:26] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #11: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/11/ [19:17:14] Yippee, build fixed! [19:17:15] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #11: 09FIXED in 28 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/11/ [19:25:02] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:32] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:08] twentyafterfour: did you read phabs new upstream change? They are now only taking bug reports and feature requests from paying users [19:39:23] https://secure.phabricator.com/conpherence/1336/#44180 [19:40:31] Sagan: yeah I am familiar with that [19:41:14] twentyafterfour: in my opinion I'd say it's a step into the wrong direction [19:42:16] they get a lot of unhelpful reports, their support forum will still funnel the valid stuff into bugs / feature requests [19:42:26] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:43:02] in my experience they are always eager to fix valid / reproducible bugs [19:43:33] and to entertain feature requests that are backed by a decent use case [19:45:04] but they get a lot of reports that are not reproducible and it takes a ton of time for them to evaluate each one / attempt to reproduce a bunch of obscure stuff [20:02:37] could be. from my view it's just: I discovered a minor bug, which is easy to fix, and just wanted to report it, and create a task like I do normally, but from what I read then, I don't have the motivation nor the pleasure to create an account at such a forum, to ask nicely then for a bugfix. then I prefer for working on other software instead [20:33:13] Sagan: you can always report it in phabricator.wikimedia.org and let us deal with upstreaming it [20:33:26] evan priestley also monitors reports in our phabricator instance [20:35:32] twentyafterfour: ok, thanks :). But for now I prefer focussing the time I have on a project were the community is more valued, so more wmf projects or something like that [20:47:24] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [21:11:40] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:31:42] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:20] (03PS1) 10Umherirrender: Add .gitattributes [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/365503 [22:09:48] (03CR) 10Umherirrender: "I have no idea how to test, if the exclude of folders works" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/365503 (owner: 10Umherirrender) [22:13:35] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:41:02] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:46:04] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [22:56:04] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:15:49] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:21:02] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:22:39] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:23:23] PROBLEM - Puppet errors on castor is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:43:37] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:53:32] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]