[02:34:54] <wmf-insecte>	 Project beta-scap-eqiad build #34943: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34943/
[02:44:57] <wmf-insecte>	 Project beta-scap-eqiad build #34944: STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34944/
[02:55:19] <wmf-insecte>	 Yippee, build fixed!
[02:55:20] <wmf-insecte>	 Project beta-scap-eqiad build #34945: FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/34945/
[03:07:44] <wmf-insecte>	 Yippee, build fixed!
[03:07:45] <wmf-insecte>	 Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #303: FIXED in 1 min 44 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/303/
[03:35:00] <wmf-insecte>	 Yippee, build fixed!
[03:35:00] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #151: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/151/
[03:40:18] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #204: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/204/
[04:11:15] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-sca-cache01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]  
[04:24:39] <wmf-insecte>	 Yippee, build fixed!
[04:24:40] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #430: FIXED in 1 hr 3 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/430/
[04:32:26] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%)  
[04:52:25] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<33.33%)  
[06:37:27] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK  
[06:56:34] <shinken-wm>	 PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[07:21:31] <shinken-wm>	 RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:06:44] <wikibugs>	 3Analytics-EventLogging, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939288 (10Amire80) 3NEW
[08:07:31] <wikibugs>	 3Release-Engineering, Continuous-Integration: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#939295 (10Amire80)
[08:24:04] <wikibugs>	 3Analytics-EventLogging, ContentTranslation-Deployments, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939308 (10Arrbee)
[08:54:44] <wikibugs>	 3Release-Engineering, Continuous-Integration: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#939332 (10QChris)
[09:57:46] <wikibugs>	 3Release-Engineering, Continuous-Integration: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#939426 (10hashar)
[10:47:52] <shinken-wm>	 PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]  
[10:52:25] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [3600.0]  
[11:11:04] <shinken-wm>	 PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[11:14:32] <wikibugs>	 3Beta-Cluster, operations: Beta servers can be badly misconfigured if mwyaml hiera backend fails - https://phabricator.wikimedia.org/T78408#939527 (10fgiunchedi) I think it is currently fail-open otherwise that creates a circular dependency between puppet and wikitech, perhaps it can be made more obvious when th...
[11:17:54] <shinken-wm>	 RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[11:36:05] <shinken-wm>	 RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0]  
[12:16:14] <wikibugs>	 3Analytics-EventLogging, ContentTranslation-Deployments, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939595 (10hashar) So looking quickly at ULS, it depends on a RL module `schema.UniversalLanguageSelector` which is not i...
[12:18:03] <grrrit-wm1>	 (03PS1) 10Hashar: ULS depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/181400 
[12:18:59] <grrrit-wm1>	 (03CR) 10jenkins-bot: [V: 04-1] ULS depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar)
[12:19:59] <wikibugs>	 3Analytics-EventLogging, ContentTranslation-Deployments, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939608 (10Amire80) As I said, AFAIK the schema in ULS is loaded correctly using the hook, with the technique described h...
[12:21:06] <grrrit-wm1>	 (03CR) 10Amire80: [C: 031] ULS depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar)
[12:29:11] <wikibugs>	 3Analytics-EventLogging, ContentTranslation-Deployments, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939610 (10hashar) The problem is the Jenkins job does not clone the EventLogging extension when preparing the tests for...
[12:30:05] <grrrit-wm1>	 (03PS2) 10Hashar: ULS depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/181400 
[12:31:39] <shinken-wm>	 PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[12:32:37] <grrrit-wm1>	 (03CR) 10Amire80: [C: 031] ULS depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar)
[12:34:17] <wikibugs>	 3operations, Continuous-Integration: [OPS] hhvm 3.3.0-20140925+wmf3 has some annoying build dependency - https://phabricator.wikimedia.org/T73413#939617 (10Joe)
[12:36:44] <grrrit-wm1>	 (03CR) 10KartikMistry: [C: 031] ULS depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar)
[12:53:54] <shinken-wm>	 PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[13:18:56] <shinken-wm>	 RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:20:28] <shinken-wm>	 PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[13:24:47] <wikibugs>	 3operations, Release-Engineering: Create phpunit test in mediawiki-config repo to validate Parsoid settings - https://phabricator.wikimedia.org/T70532#939668 (10Dzahn) based on: phpunit test = CI team, mw-config = Platform eng. team, Parsoid = Services team can i claim that it's not an Operations team ticket?
[13:27:06] <wikibugs>	 3Release-Engineering: Create phpunit test in mediawiki-config repo to validate Parsoid settings - https://phabricator.wikimedia.org/T70532#939678 (10Dzahn)
[13:28:35] <grrrit-wm1>	 (03CR) 10Hashar: "Updated jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar)
[13:31:05] <grrrit-wm1>	 (03CR) 10Hashar: [C: 032] "And that fixed ULS job :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar)
[13:32:05] <wikibugs>	 3ContentTranslation-Deployments, Analytics-EventLogging, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939691 (10Amire80) Looks good to me now. Thank you.
[13:32:18] <wikibugs>	 3ContentTranslation-Deployments, Analytics-EventLogging, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939692 (10Amire80) a:3hashar
[13:32:29] <wikibugs>	 3ContentTranslation-Deployments, Analytics-EventLogging, Continuous-Integration: UniversalLanguageSelector patches are blocked by voting schema tests - https://phabricator.wikimedia.org/T85124#939693 (10Amire80) 5Open>3Resolved
[13:35:23] <grrrit-wm1>	 (03Merged) 10jenkins-bot: ULS depends on EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/181400 (owner: 10Hashar)
[13:40:25] <shinken-wm>	 RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:41:03] <wikibugs>	 3operations, Beta-Cluster: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#939712 (10Dzahn)
[13:42:08] <wikibugs>	 3operations, Beta-Cluster: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#835083 (10Dzahn) renamed task. the patch changes this for ALL mediawiki hosts. i don't see how it just affected beta or just trusty. in production ALL apaches also have the wrong uid. also, the goal should...
[14:07:48] <shinken-wm>	 PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]  
[14:37:49] <shinken-wm>	 RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[15:09:04] <wikibugs>	 3operations, Beta-Cluster: Beta servers can be badly misconfigured if mwyaml hiera backend fails - https://phabricator.wikimedia.org/T78408#939862 (10Joe) I agree with yuvi, it should fail if the wikitech fetch fails for a request timeout. This would however cause a lot of unwanted failures.  To correctly troubl...
[15:48:42] <shinken-wm>	 PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[15:52:08] <shinken-wm>	 PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[16:13:38] <shinken-wm>	 RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0]  
[16:17:06] <shinken-wm>	 RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0]  
[16:19:10] <wikibugs>	 3Quality-Assurance: Quality Assurance/Browser testing/Setup instructions is out of date - https://phabricator.wikimedia.org/T74732#939968 (10Cmcmahon) a:5Ckoerner>3Cmcmahon
[16:45:08] <shinken-wm>	 PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[16:46:22] <shinken-wm>	 PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[16:56:14] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [3600.0]  
[17:05:03] <wikibugs>	 3operations, Beta-Cluster: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#940108 (10bd808) >>! In T78076#939712, @Dzahn wrote: > renamed task. the patch changes this for ALL mediawiki hosts. i don't see how it just affected beta or just trusty. in production ALL apaches also have...
[21:11:19] <wikibugs>	 3Release-Engineering, Continuous-Integration: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions (tracking) - https://phabricator.wikimedia.org/T75521#940508 (10hashar)
[21:11:20] <wikibugs>	 3Continuous-Integration, operations: [OPS] hhvm 3.3.0-20140925+wmf3 has some annoying build dependency - https://phabricator.wikimedia.org/T73413#940505 (10hashar) 5Open>3Resolved a:3hashar Fixed by @joe with HHVM package 3.3.1+dfsg1-1+wm1 . The changelog has:  ``` hhvm (3.3.1+dfsg1-1+wm1) UNRELEASED; urge...
[21:32:20] <wmf-insecte>	 Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #239: FAILURE in 19 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/239/
[21:38:00] <wmf-insecte>	 Project beta-scap-eqiad build #35056: FAILURE in 4 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35056/
[21:43:32] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #359: FAILURE in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/359/
[21:45:37] <wmf-insecte>	 Project beta-scap-eqiad build #35057: STILL FAILING in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35057/
[22:14:27] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #361: FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/361/
[22:19:30] <wmf-insecte>	 Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #78: FAILURE in 5 min 2 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/78/
[22:20:26] <wmf-insecte>	 Yippee, build fixed!
[22:20:26] <wmf-insecte>	 Project beta-scap-eqiad build #35058: FIXED in 25 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35058/
[22:40:08] <wmf-insecte>	 Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #359: FAILURE in 10 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/359/
[23:03:58] <wikibugs>	 3Quality-Assurance: Quality Assurance/Browser testing/Setup instructions is out of date - https://phabricator.wikimedia.org/T74732#940794 (10Cmcmahon) 5Open>3Resolved I made significant changes to the whole page. It's a lot nicer now, and more accurate also.
[23:16:03] <wmf-insecte>	 Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #354: FAILURE in 35 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/354/
[23:49:01] <wikibugs>	 3Mobile-Web, MediaWiki-General-or-Unknown, Continuous-Integration: QUnit job unable to collect logs, test still has active http requests writing to them - https://phabricator.wikimedia.org/T78590#940924 (10Jdlrobson) I suspect it might not be an ajax request but a JavaScript timeout somewhere that might be causi...