[00:28:29] 3Wikimedia Labs / 3deployment-prep (beta): beta labs job queue stuck (again) - 10https://bugzilla.wikimedia.org/70374 (10Kunal Mehta (Legoktm)) 3NEW p:3Unprio s:3major a:3None http://en.wikipedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/ZFilipin_%28WMF%29 had a bunch of jobs queued that wer... [00:28:56] 3Wikimedia Labs / 3deployment-prep (beta): beta labs job queue stuck (again) - 10https://bugzilla.wikimedia.org/70374 (10Kunal Mehta (Legoktm)) p:5Unprio>3High [01:32:27] 3Wikimedia Labs / 3deployment-prep (beta): beta labs job queue stuck (again) - 10https://bugzilla.wikimedia.org/70374#c1 (10Bryan Davis) Manually started jobrunner service on deployment-jobrunner01. $ mwscript showJobs.php --wiki=enwiki --group Flow\Jobs\WatchTitle: 0 queued; 1422 claimed (1422 active, 0 a... [01:44:12] 3Wikimedia Labs / 3deployment-prep (beta): beta labs job queue stuck (again) - 10https://bugzilla.wikimedia.org/70374#c2 (10Bryan Davis) $ tail -5000 /var/log/mediawiki/jobrunner.log|grep 'Fatal error'|grep 'has no version entry'|awk '{print $9}'|sort|uniq -c 52 `afwikibooks`. 51 `afwikiquote`.... [08:26:42] (03PS1) 10Tobias Gritschacher: Add Job to run ResourcesTest for WikibaseRepo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 [08:27:12] (03CR) 10jenkins-bot: [V: 04-1] Add Job to run ResourcesTest for WikibaseRepo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 (owner: 10Tobias Gritschacher) [08:29:32] (03PS2) 10Tobias Gritschacher: Add Job to run ResourcesTest for WikibaseRepo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 [10:12:13] (03CR) 10Aude: [C: 031] Add Job to run ResourcesTest for WikibaseRepo [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 (owner: 10Tobias Gritschacher) [10:45:34] addshore: just checking, are we pairing today? (in an hour) [10:45:48] addshore: sorry, wanted to ask aharoni that :) [10:45:52] :D [10:45:57] :D [10:46:14] http://www.damnyouautocorrect.com/ [12:33:32] (03CR) 10JanZerebecki: "I don't see the tests that would run. That comes up empty for me:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 (owner: 10Tobias Gritschacher) [12:41:22] (03CR) 10Addshore: [C: 04-1] "Adding this as a separate job will mean about 2 mins of overhead for running a test that probably takes 2 seconds" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 (owner: 10Tobias Gritschacher) [12:43:06] (03CR) 10Addshore: "It might be an idea to instead add a group to ResourcesTest in core that can be added to the list of group we run" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 (owner: 10Tobias Gritschacher) [15:00:35] (03CR) 10JanZerebecki: [C: 04-1] "If the goal is to run some tests from core then there is no way around this (group or not), because currently none of the tests from core " [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158338 (owner: 10Tobias Gritschacher) [15:12:44] (03PS4) 10Zfilipin: WIP Trying to create IE11 Jenkins job for VisualEditor [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157164 (https://bugzilla.wikimedia.org/70126) [15:12:49] (03PS4) 10Zfilipin: browsertests: Enable VE on IE and Safari; remove Chrome on Windows XP [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157480 (https://bugzilla.wikimedia.org/70126) (owner: 10Jforrester) [15:13:28] (03CR) 10jenkins-bot: [V: 04-1] WIP Trying to create IE11 Jenkins job for VisualEditor [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157164 (https://bugzilla.wikimedia.org/70126) (owner: 10Zfilipin) [15:24:21] (03CR) 10Zfilipin: "recheck" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157164 (https://bugzilla.wikimedia.org/70126) (owner: 10Zfilipin) [15:32:39] (03PS1) 10Manybubbles: Jenkins job for elasticsearch-swift plugin [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 [15:34:13] (03CR) 10jenkins-bot: [V: 04-1] Jenkins job for elasticsearch-swift plugin [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 (owner: 10Manybubbles) [15:35:11] (03CR) 10Manybubbles: "I'm not sure what that failure means....." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 (owner: 10Manybubbles) [15:36:44] (03CR) 10Manybubbles: "I also tried adding JenkinsBot with Submit privileges to https://gerrit.wikimedia.org/r/#/admin/projects/search/repository-swift,access bu" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 (owner: 10Manybubbles) [15:41:11] (03PS5) 10Zfilipin: WIP Trying to create IE11 Jenkins job for VisualEditor [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157164 (https://bugzilla.wikimedia.org/70126) [15:41:35] (03CR) 10jenkins-bot: [V: 04-1] WIP Trying to create IE11 Jenkins job for VisualEditor [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157164 (https://bugzilla.wikimedia.org/70126) (owner: 10Zfilipin) [15:46:21] (03CR) 10Zfilipin: WIP Trying to create IE11 Jenkins job for VisualEditor (033 comments) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/157164 (https://bugzilla.wikimedia.org/70126) (owner: 10Zfilipin) [15:55:27] 3Wikimedia Labs / 3deployment-prep (beta): beta labs job queue stuck (again) - 10https://bugzilla.wikimedia.org/70374#c3 (10Bryan Davis) Tried to clean up bad jobs manually: redis 127.0.0.1:6379> keys *:jobqueue:LocalRenameUserJob:l-* 1) "nlwikiquote:jobqueue:LocalRenameUserJob:l-unclaimed" 2) "amwikiquot... [16:06:12] 3Wikimedia Labs / 3deployment-prep (beta): beta labs job queue stuck (again) - 10https://bugzilla.wikimedia.org/70374#c4 (10Bryan Davis) 5NEW>3RESO/FIX W00t figured it out. There is a special hash for the new jobrunner that tracks what queues to try and process: redis 127.0.0.1:6379> hkeys "jobqueue:agg... [16:20:41] (03CR) 10Chad: "recheck" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 (owner: 10Manybubbles) [16:22:51] (03CR) 10Chad: "I'm confused. I get no problems at all doing this locally." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 (owner: 10Manybubbles) [16:25:15] (03PS2) 10Chad: Jenkins job for elasticsearch-swift plugin [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 (owner: 10Manybubbles) [16:43:33] (03PS3) 10Chad: Jenkins job for elasticsearch-swift plugin [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/158396 (owner: 10Manybubbles)