[00:00:58] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 10releng-201516-q3, 07WorkType-NewFunctionality: [keyresult] Migrate as many misc CI jobs as possible to Nodepool - https://phabricator.wikimedia.org/T119140#2588916 (10Paladox) [00:00:59] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling: Track what tests we have left to convert to nodepool - https://phabricator.wikimedia.org/T136249#2588915 (10Paladox) [00:01:31] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling, 13Patch-For-Review, 07WorkType-NewFunctionality: Migrate mwext-testextension* jobs to nodepool - https://phabricator.wikimedia.org/T137199#2588918 (10Paladox) [00:01:33] 10Continuous-Integration-Config, 05Continuous-Integration-Scaling: Track what tests we have left to convert to nodepool - https://phabricator.wikimedia.org/T136249#2328123 (10Paladox) [00:20:02] Project beta-update-databases-eqiad build #10919: 04STILL FAILING in 0.89 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10919/ [00:38:43] 10Continuous-Integration-Infrastructure, 07Zuul: Fix zuul-cloner so that it checkout the ref to the local branch instead of a detached one - https://phabricator.wikimedia.org/T141607#2588928 (10Paladox) I think this may be fixed with rCIZU84c4f940000f731265416c7c0e4b482cd1a713da [00:57:18] 10Continuous-Integration-Infrastructure: Allow people with +2 rights to trigger recheck - https://phabricator.wikimedia.org/T144113#2588929 (10whym) [01:06:23] 10Continuous-Integration-Infrastructure: Allow people with +2 rights to trigger recheck - https://phabricator.wikimedia.org/T144113#2588929 (10Paladox) This won't be possible since we have a whitelist for users allowed to recheck. But as we move to nodepool we will get rid of the whitelist and open it to everyone. [01:20:01] Project beta-update-databases-eqiad build #10920: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10920/ [01:34:05] 10Beta-Cluster-Infrastructure, 06Labs: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2588952 (10mmodell) a:0520after4>03Andrew @andrew: `puppet::self` is very prone to race conditions and it needs to be completely refactored. I don't know if this was caused by {8a... [01:34:47] andrewbogott: feel free to revert 8a4cc8790f4546a22fdad2cc7a4dec412a831e46, it was a blind shot at fixing a race condition and it certainly could have caused other races. The whole puppet::self setup is fragile and nasty. [02:20:01] Project beta-update-databases-eqiad build #10921: 04STILL FAILING in 0.88 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10921/ [03:20:01] Project beta-update-databases-eqiad build #10922: 04STILL FAILING in 0.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10922/ [04:20:01] Project beta-update-databases-eqiad build #10923: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10923/ [05:20:01] Project beta-update-databases-eqiad build #10924: 04STILL FAILING in 0.82 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10924/ [06:20:01] Project beta-update-databases-eqiad build #10925: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10925/ [07:20:01] Project beta-update-databases-eqiad build #10926: 04STILL FAILING in 0.77 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10926/ [07:22:19] 10Beta-Cluster-Infrastructure, 06Labs: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2589029 (10hashar) Keep rOPUP8a4cc8790f ! Else we are back to T132689 which has was quite annoying. [07:38:22] 10Continuous-Integration-Infrastructure, 07Zuul: Fix zuul-cloner so that it checkout the ref to the local branch instead of a detached one - https://phabricator.wikimedia.org/T141607#2589031 (10hashar) >>! In T141607#2588928, @Paladox wrote: > I think this may be fixed with rCIZU84c4f940000f731265416c7c0e4b482... [07:42:19] (03CR) 10Hashar: "Eek! Thank you to have caught and fixed that one." [integration/docroot] - 10https://gerrit.wikimedia.org/r/298275 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [07:48:26] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#2589035 (10hashar) For beta I guess we can stick to have API and Web to be served by the same boxes. T125976 is about having the various maintenance script / cron to run o... [07:49:21] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#2589037 (10hashar) [08:05:09] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 587 bytes in 0.009 second response time [08:06:17] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 589 bytes in 0.010 second response time [08:20:01] Project beta-update-databases-eqiad build #10927: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10927/ [08:43:42] (03PS1) 10Hashar: [analytics/reportupdater] add tox [integration/config] - 10https://gerrit.wikimedia.org/r/307113 (https://phabricator.wikimedia.org/T144119) [08:44:38] 10Continuous-Integration-Config, 10Analytics, 13Patch-For-Review: Add test runner and CI configuration to analytics/reportupdater - https://phabricator.wikimedia.org/T144119#2589109 (10hashar) [08:44:52] 10Continuous-Integration-Config, 10Analytics, 13Patch-For-Review: Add test runner and CI configuration to analytics/reportupdater - https://phabricator.wikimedia.org/T144119#2589092 (10hashar) p:05Triage>03Normal [09:09:59] 10Continuous-Integration-Infrastructure, 07Zuul: Fix zuul-cloner so that it checkout the ref to the local branch instead of a detached one - https://phabricator.wikimedia.org/T141607#2589132 (10Paladox) @hashar oh would you be able to create a debian patch for that please? [09:20:01] Project beta-update-databases-eqiad build #10928: 04STILL FAILING in 0.82 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10928/ [10:20:01] Project beta-update-databases-eqiad build #10929: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10929/ [11:20:01] Project beta-update-databases-eqiad build #10930: 04STILL FAILING in 0.83 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10930/ [12:20:01] Project beta-update-databases-eqiad build #10931: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10931/ [13:20:01] Project beta-update-databases-eqiad build #10932: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10932/ [14:06:44] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [14:20:01] Project beta-update-databases-eqiad build #10933: 04STILL FAILING in 0.81 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10933/ [15:20:02] Project beta-update-databases-eqiad build #10934: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10934/ [15:22:16] Those errors are todo with mobilefrontend [15:22:27] Tracked at https://phabricator.wikimedia.org/T143974 [15:28:16] paladox: are you able to trigger the beta-update-databases job manually? [15:28:26] FlorianSW nope [15:34:11] FlorianSW i have reproducable steps [15:34:30] Install MobileFrontend from master branch [15:34:40] onto a 1.28 wmf 16 mw [15:34:50] then add [15:34:51] require_once "$IP/extensions/MobileFrontend/MobileFrontend.php"; [15:34:51] $wgMFAutodetectMobileView = true; [15:34:55] to localsettings.php [15:35:05] and then go to your wiki, it should show a 500 error [15:36:11] paladox: no it shouldn't. From what you say, let me try (however, this doesn't seem to be the error described in the task, as vieweing a page isn't affected) [15:36:28] FlorianSW see https://en.random-wikisaur.tk/ [15:36:45] ok, and what does the error log say? [15:37:01] my local mediawiki installation works fine with 1.28.0-wmf.16 and MF master [15:37:04] https://phabricator.wikimedia.org/P3902 [15:37:26] FlorianSW ^^ [15:37:45] Ignore _ since i was testing something [15:38:20] can you paste your LocalSettings.php (without passwords, of course) [15:38:29] Yes [15:40:42] FlorianSW https://phabricator.wikimedia.org/P3903 [15:46:08] hmm, I can reproduce the problem, if I set $wgMainCacheType = CACHE_ANYTHING; [15:46:43] Oh [15:47:07] Maybe it is a mw problems and not a mobilefrontend problem? [15:47:47] the question before that is: as far as I know, $wgMainCacheType is set to memcached in wmf-production. [15:47:55] I'm not sure what is set in labs [15:48:48] Oh [15:50:36] https://github.com/wikimedia/mediawiki/blob/4737ea0412783ee45f5522c6053717ad0ea4e8a6/includes/Services/ServiceContainer.php#L183 [15:51:22] https://github.com/wikimedia/mediawiki/blob/4737ea0412783ee45f5522c6053717ad0ea4e8a6/includes/Services/ServiceContainer.php#L241 [15:51:51] yes, yes, I know that file. I currently amend the cache settings of wmf-labs [15:51:56] (beta labs) [15:52:34] Ok [15:59:26] hmm, can't reproduce with memcached so far :/ [15:59:40] Oh [16:20:01] Project beta-update-databases-eqiad build #10935: 04STILL FAILING in 0.87 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10935/ [16:41:59] paladox: uploaded a change that should fix the problem. Could you please check in your environment, too? :) [16:42:13] FlorianSW ok, will do that now [16:42:18] and thanks for looking at a fix [16:44:03] FlorianSW that fixes the problem [16:44:07] thanks for fixing it [16:44:23] np, let's see if this is a desired behaviour :) [16:45:06] Ok, i hope they merge it :) [17:20:01] Project beta-update-databases-eqiad build #10936: 04STILL FAILING in 0.78 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10936/ [18:05:28] FlorianSW hi, regarding https://phabricator.wikimedia.org/T98347 that is still a problem and i can still reproduce. If your interested in helping me track down the problem since i think it may be mw side not the extension i am trying to convert. [18:05:47] I can upload another patch which you can install on your localmachine for the extension [18:05:48] ? [18:20:02] Project beta-update-databases-eqiad build #10937: 04STILL FAILING in 0.88 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10937/ [18:44:38] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:42] RECOVERY - Puppet run on integration-slave-precise-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [18:53:27] RECOVERY - Puppet run on integration-slave-precise-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [19:20:01] Project beta-update-databases-eqiad build #10938: 04STILL FAILING in 0.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10938/ [20:20:01] Project beta-update-databases-eqiad build #10939: 04STILL FAILING in 0.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10939/ [20:55:22] Project selenium-Wikidata » firefox,test,Linux,contintLabsSlave && UbuntuTrusty build #100: 04FAILURE in 2 hr 5 min: https://integration.wikimedia.org/ci/job/selenium-Wikidata/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/100/ [21:04:54] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2589664 (10Florian) [21:16:25] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T142117#2589678 (10hashar) I am confirming that I am going to rid the train this week [21:20:02] Project beta-update-databases-eqiad build #10940: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10940/ [21:58:21] Yippee, build fixed! [21:58:22] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #136: 09FIXED in 6 min 20 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/136/ [22:21:52] Project beta-update-databases-eqiad build #10941: 04STILL FAILING in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10941/ [23:12:06] PROBLEM - Host deployment-poolcounter01 is DOWN: CRITICAL - Host Unreachable (10.68.19.181) [23:21:00] Yippee, build fixed! [23:21:00] Project beta-update-databases-eqiad build #10942: 09FIXED in 58 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/10942/ [23:53:49] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:54:51] PROBLEM - SSH on integration-slave-trusty-1012 is CRITICAL: Server answer