[10:28:05] 02/02/2012 - 10:28:05 - Creating a home directory for schmir at /export/home/pediapress/schmir [10:28:12] 02/02/2012 - 10:28:11 - Creating a home directory for schmir at /export/home/bastion/schmir [10:29:05] 02/02/2012 - 10:29:05 - Updating keys for schmir [10:29:12] 02/02/2012 - 10:29:11 - Updating keys for schmir [11:46:40] New patchset: Mark Bergsma; "Merge branch 'production' into test" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2195 [11:49:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2195 [11:49:44] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2195 [12:03:04] New patchset: Mark Bergsma; "Fix swift lvs service in labs" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2197 [12:03:30] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2197 [12:03:31] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2197 [12:16:33] New patchset: Mark Bergsma; "Make lvs::realserver support hashes as well as arrays for service IPs" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2198 [12:16:51] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2198 [12:19:31] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2198 [12:19:32] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2198 [12:27:50] New patchset: Mark Bergsma; "Make lvs::realserver a parameterized class" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2199 [12:28:09] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2199 [12:28:20] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2199 [12:28:20] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2199 [12:28:31] I can explain... [12:33:00] New patchset: Mark Bergsma; "Fix reassignment of realserver_ips" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2200 [12:33:18] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2200 [12:33:34] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2200 [12:33:34] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2200 [12:39:46] !log incubator Deleting instances incubator-bots and incubator-bots3 due to complains, keeping bots2 due to it hosting real incubator bots [12:39:54] Logged the message, Master [12:44:20] we would love to hear an explanation by email [12:48:47] New patchset: Mark Bergsma; "Simplify role::cache::squid:: by passing a hash to lvs::realserver" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2201 [12:49:03] New patchset: Mark Bergsma; "Make all $realm references in lvs.pp fully qualified" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2202 [12:49:19] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/2201 [12:50:29] New patchset: Mark Bergsma; "Fix syntax errors" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2203 [12:50:47] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2203 [12:51:05] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2203 [12:51:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2201 [12:51:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2201 [12:51:28] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2201 [12:51:45] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2202 [12:51:46] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2203 [12:51:46] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2202 [12:59:00] mark: you know how that nagios bot work? [12:59:10] because it stopped working [12:59:23] when I do echo "test" >> ircfile it doesn't echo [12:59:44] I remember Ryan copied it from prod, but I don't really know how it work [13:00:16] no idea [13:00:21] oh, maybe I know [13:00:51] this is a test of nagios - petr bena [13:01:02] heh [13:01:10] I quieted it [13:01:14] RECOVERY Total Processes is now: OK on master master output: PROCS OK: 94 processes [13:01:54] RECOVERY dpkg-check is now: OK on master master output: All packages OK [13:02:24] RECOVERY Current Users is now: OK on master master output: USERS OK - 0 users currently logged in [13:03:21] :o [13:03:44] RECOVERY Current Load is now: OK on master master output: OK - load average: 0.20, 0.07, 0.02 [13:23:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [13:23:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [13:23:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [13:23:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [13:24:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [13:24:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [13:53:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [13:53:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [13:53:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [13:53:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [13:54:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [13:54:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [14:23:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [14:23:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [14:23:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [14:23:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [14:24:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [14:24:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [14:53:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [14:53:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [14:53:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [14:53:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [14:54:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [14:54:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [15:23:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [15:23:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [15:23:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [15:23:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [15:24:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [15:24:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [15:46:24] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 33% free memory [15:53:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [15:53:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [15:53:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [15:53:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [15:54:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [15:54:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [16:23:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [16:23:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [16:23:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [16:23:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [16:24:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [16:24:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [16:30:11] Ryan_Lane: hey [16:30:19] is there a reason for mx small to have 10 gb /mnt [16:30:28] I mean, I don't care [16:30:34] was it supposed to be larger? [16:30:36] but it eats space :) we don't need it [16:30:45] it wasn't supposed to be there [16:30:48] oh. well, it only eats space if people use it [16:30:52] ah, ok [16:31:01] it will be probably empty on all instance [16:31:19] well, maybe not ;) [16:31:20] because apache is installed in / and data are on nfs [16:31:30] definitely on all deployment instances [16:31:43] not a problem [16:31:45] ok [16:32:57] + I was thinking of creating ether for the upcoming labs meeting [16:34:09] !labsconf is http://etherpad.wikimedia.org/LabsIrcConf [16:34:09] Key was added! [16:37:27] Ryan_Lane: feel free to insert more topics there [16:38:02] petan|wk: have you applied for a wikimania scholarship? [16:38:08] yes [16:38:21] ah. good [16:38:33] was going to recommend that you do if you had not. heh [16:38:34] many Petr's did as I was told :) [16:38:46] :D [16:38:54] heh [16:53:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [16:53:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [16:53:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [16:53:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [16:54:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [16:54:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [17:23:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [17:23:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [17:23:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [17:23:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [17:24:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [17:24:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [17:33:55] New patchset: Mark Bergsma; "Merge remote-tracking branch 'origin/production' into test" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2227 [17:34:14] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2227 [17:36:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2227 [17:36:05] Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2227 [17:53:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [17:53:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [17:53:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [17:53:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [17:54:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [17:54:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [17:55:23] !log deployment Installed imagemagick to dbdump [17:55:23] deployment is not a valid project. [17:55:29] !log deployment-dbdump Installed imagemagick to dbdump [17:55:29] deployment-dbdump is not a valid project. [18:23:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [18:23:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [18:23:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [18:23:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [18:24:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [18:24:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [18:53:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [18:53:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [18:53:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [18:53:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [18:54:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [18:54:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [19:23:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [19:23:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [19:23:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [19:23:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [19:24:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [19:24:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [19:45:55] hey oy :) [19:46:09] does anyone know if we can access our git repo as anonymous users [19:46:18] something like git clone http://gerrit.wikimedia.org:29418/integration/jenkins [19:46:30] Should be possible [19:46:53] Of course the Man That Knows Everything About Labs is on a plane [19:47:44] good to know you have IRC access while you are flying around :-D [19:48:21] <^demon> https://gerrit.wikimedia.org/r/p/integration/jenkins.git [19:48:44] hashar: Not me, Ryan [19:48:51] ^demon: you rocks [19:49:07] well you all rocks [19:49:12] and are awesome etc... [19:49:31] I am tired of this new jobs cause there is too many popstars around and that makes it hard to shine [19:50:09] * ^demon feels the WikiLove [19:50:40] <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 <3 [19:51:51] <^demon> RoanKattouw: Btw, I found the best picture ever that shows gerrits workflow :D [19:51:53] <^demon> http://source.android.com/source/life-of-a-patch.html [19:53:34] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [19:53:34] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [19:53:44] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [19:53:44] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [19:54:04] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [19:54:06] Looks reasonable [19:54:38] The terminology it uses is a bit weird, there's some weirdness in the verified/reviewed interaction, and it's missing automated integration tests etc [19:54:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [19:55:08] * RoanKattouw wants to have all of those parts of the MW dev workflow documented some day [19:55:51] <^demon> [[mw:Git/Workflow]] was started yesterday [19:55:56] <^demon> Feel free to expand :) [19:56:02] https://integration.mediawiki.org/ci/job/MediaWiki%20-%20GIT%20testing/1/console [19:56:09] got a jenkins job fetching your repo chad [19:56:25] today I have worked on taking other the jenkins jobs repo [19:56:46] <^demon> We really need to make puppet check out those jobs. [19:56:48] will have to write something on mw.org [19:56:58] Huh, you don't need that, do you? [19:57:08] The Gerrit integration plugin does that sort of thing for you [19:57:17] <^demon> I mean the job config files. [19:57:17] ^demon: the idea would be that jenkins fetches its configuration :D [19:57:28] I meant the git checkout job [19:57:36] <^demon> Oh yeah, jenkins does that itelf. [19:57:37] I don't want to depends upon puppet, cause it needs an op to merge the change and then manually run puppet [19:57:41] It's unfortunate that our Jenkins work is decentralized [19:58:11] decentralized ? woot do you mean? [19:58:14] <^demon> hashar: Then how do you want to pull the job config files from the repo to /var/lib/jenkins/jobs? [19:58:20] <^demon> If not via puppet [19:58:44] by running a jenkins job? [19:58:51] not sure how to make it a dependency for all jobs though [19:59:12] <^demon> Ew. [19:59:16] You could fetch the jobs from the jobs repo using puppet, right? [19:59:35] Then if you change a job, you change it in the jobs repo, not in the puppet repo. You wouldn't have to change anything in puppet, puppet just make sure the jobs are up-to-date [19:59:38] I can just do a jenkins job that fetch the jobs configuration [19:59:40] <^demon> there's no way to point file{} directives at other repos yet. [19:59:45] on a 1 minute schedule or so [19:59:47] <^demon> Only the local FS or the puppet repo. [20:00:05] or just use a post commit hook in the integration/jenkins repo to notify jenkins that it needs to build the job that fetch the job conf [20:00:07] something like that [20:00:16] Well [20:00:33] I want to move the post-merge hook and other hooks into Jenkins as well :) [20:00:37] <^demon> Or, just do it via puppet like I suggested ~2 months ago :) If you do it generically enough all you'd need to do in the puppet repo is update the commit hash you want to pull. [20:00:57] idealy, gerrit should submit a job request to jenkins and wait for its result [20:01:04] Hm [20:01:09] Can't you just pull the most recent version? [20:01:21] <^demon> Yeah I don't think that'll fly with ops. [20:01:32] hashar: That's all possible with the Gerrit Trigger Plugin, which I installed in my labs VM :) [20:01:34] Hm, right [20:01:43] OR [20:01:45] Well ops either needs to review the jenkins jobs or they don't [20:01:53] If they do, they should be in the puppet repo [20:02:03] If they don't, puppet might as well fetch the latest one blindly [20:02:17] <^demon> Well, they don't need to review them. But the idea of "just pull whatever the hell HEAD is" makes me anxious. [20:02:48] Hmm, true, puppet also doesn't have automatic deployment currently [20:03:01] That's a manual update from the shell right now [20:03:09] if we really want to do continuous integration, we could even have jenkins deploy stuff on the live cluster [20:03:16] * hashar *grins evily* [20:03:22] <^demon> There's a plugin for that. [20:03:24] Eww, no [20:03:25] <^demon> But it scares me ;-) [20:03:35] I want to do lots of cool stuff in Jenkins [20:03:37] But not that [20:03:47] we will do it [20:03:49] eventually [20:03:52] it is all a matter of time [20:03:56] <^demon> Direct deployments from jenkins? [20:04:01] <^demon> I'm going to veto that one too. [20:04:08] just like we are no naming our server db53 instead of larousse or pliny :D [20:05:04] we could well push change for review in gerrit [20:05:12] gerrit submit a verification job to jenkins [20:05:21] change is then approved / validated [20:05:27] gerrit send a notification to jenkins [20:05:29] jenkins deploy [20:05:41] profit [20:05:54] (oh no we are a charity … so s/profit/enjoy/ ) [20:08:47] <{98c0-4d-11-9a66> <-- hashar : will hopefully write some scenario next week [20:08:56] <{98c0-4d-11-9a66> for now, heading to the monthly LUG meeting [20:09:03] hashar: I would like to see an automatic deployment to a labs env upon commit (or approved commit, or whatever). looking forward to it. [20:09:42] After starting a instance from labsconsole, how am I supposed to become root on that? [20:12:46] sudo should just work [20:13:00] chrismcmahon: Hmm, yeah, that's a good idea [20:13:00] RoanKattouw: it asks for a password [20:13:09] So type in your password [20:13:35] ok. I thought that was just for the web interface [20:13:49] I thought so too, but apparently not [20:13:58] thanks [20:14:14] RoanKattouw: I think it's a good idea. And not too long after that, a small suite of UI/browser tests also hooked into the Jenkens deploy job. But that's a little ways out. [20:14:32] Aye [20:14:38] In the short term, I have some evil plans for Jenkins [20:14:55] Which will probably^W hopefully be executed by ^demon and hashar instead, because I'm busy [20:14:57] * RoanKattouw finds link [20:15:13] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Jenkins [20:16:14] <^demon> That's right Roan, delegate ;-) [20:16:28] heh [20:16:47] One way-to-busy guy delegating to another [20:16:50] *too [20:17:18] Once the dust has settled on 1.19 review and my move I'll hopefully have time to poke at this, and at the git migration project as a whole [20:18:00] RoanKattouw: I discussed this briefly today with robla, git conversion is the clear next step. [20:18:06] Aye [20:18:16] But much of this Jenkins stuff is a prerequisite for the git conversion [20:18:38] :-) that's pretty circular [20:19:06] Not really [20:19:19] We can't really move to git if we don't have things like IRC notifications for commits [20:19:41] And we shouldn't move to git if we don't have Jenkins testing set up, because we'd be backpedaling on the CI front in that case [20:19:54] The other things such as BZ integration and lint checks are extras, I guess [20:20:54] ok, makes sense. I was thinking the svn->git move is the hardest part, but I'm a n00b. [20:22:35] PROBLEM dpkg-check is now: CRITICAL on bob bob output: DPKG CRITICAL dpkg reports broken packages [20:22:49] <^demon> Converting a svn repo to git is easy. [20:22:53] <^demon> doing it properly is harder ;-) [20:23:05] :-) [20:23:20] Converting the repo isn't the biggest problem, it's migrating the infrastructure around it [20:23:25] Well, strike that [20:23:35] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [20:23:35] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [20:23:39] Infrastructure is a bigger problem, but probably the biggest problem is just gonna be people [20:23:45] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [20:23:45] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [20:24:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [20:24:45] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [20:27:16] roan: i see on your jenkins to do list: write lint job for python [20:27:25] i think i might be able to pull that off [20:27:32] if you give me some starting points [20:27:59] It's not really specific to Python, I can probably just use pep8 [20:28:20] even though it's a small task, i can take it away from you :) [20:29:29] Do you have a labs account? [20:30:07] pyflakes is quite good [20:30:12] uhhh of course! [20:30:20] OK [20:30:23] What's your user name? [20:30:26] Diederik [20:30:27] I'll add you to the project [20:30:34] Or... maybe I won't [20:30:40] up to you :D [20:30:40] You should be able to access it without that [20:30:52] access what? [20:30:56] Do you have FoxyProxy or some other proxy thing set up? [20:30:58] The Jenkins VM [20:31:09] no, that doesn't work for me [20:31:28] how long does it take to boot an instance? https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=pediapress&instanceid=i-0000012c [20:31:47] cant connect to it: ssh: connect to host 10.4.0.88 port 22: No route to host [20:32:06] If you didn't check any puppet classes, it should boot in a few minutes tops [20:32:35] PROBLEM host: bob is DOWN address: bob CRITICAL - Host Unreachable (bob) [20:32:40] diederik: Have you tried the setup instructions at https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy ? [20:32:50] damn bob is down [20:33:12] gonna kill him. [20:33:20] i tried proxycommand ssh option, that didn't work [20:33:24] didn't try socks [20:34:18] socks works fine for me [20:34:27] You just install a Firefox extension [20:34:34] Fair warning: the config file they give you is broken [20:34:47] Rusty helped me figure that out at the SF hackathon [20:37:12] what is the name of the jenkins vm [20:38:02] jenkins2 [20:38:21] Let me get you the URL to the PHP lint jo [20:38:50] http://jenkins2.pmtpa.wmflabs:8080/job/lint/configure [20:39:01] There is a text box with a bash script somewhere [20:39:21] (I really need to rewrite that in PHP so it can use parsekit, that'll be much faster than invoking php -l on each file individually) [20:39:39] i see it [20:40:01] We need something like that but for Python [20:40:03] or pep8 [20:40:16] It doesn't really matter what language it's written in, as long as it returns the same XML format [20:41:07] okay [20:41:27] is there some documentation for this? [20:41:48] actually it's quite straightforward, right? [20:42:22] shall i write the script in python as well? or bash? [20:44:22] You can write it in whatever language you want [20:44:49] I'm not sure if it'll just take #! /usr/bin/python there , you might try [20:44:56] Or you can write the .py file and give it to me [20:45:46] output to a file or stdout? [20:45:52] stdout [20:45:57] ok [20:46:15] I might decide to change the output format later but I think this is fine for now [20:46:24] And changing the output format won't be hard anyway [20:46:30] basically iterate over all py files in patchset, apply pep, output results in xml format to stdout [20:46:33] is that right? [20:47:11] is there a project on git where to push this script to? [20:48:41] Not that I know of [20:48:45] Well, there's a jenkins thing somewhere [20:48:48] But I haven't touched it [20:48:54] Yeah, pretty much [20:49:02] Be sure to mark success/failure correctly as well [20:49:19] ok [20:49:40] integration/jenkins [20:49:40] maybe we should have a git repo for all the jenkins stuff? [20:50:15] We do [20:50:28] git clone ssh://gerrit.wikimedia.org:29418/integration/jenkins.git [20:51:29] excellent [20:53:35] shall i create a new folder MediaWiki-pep? [20:53:44] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [20:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [20:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [20:53:54] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [20:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [20:54:30] RoanKattouw: where on wikitech.wm.o or mediawiki.org or whatever should there be a link to that git repo (gerrit.wikimedia.org:29418/integration/jenkins.git)? [20:54:49] diederik: Yes [20:54:49] RoanKattouw: (basically if you don't have an opinion I will fling it somewhere) [20:54:57] I have no idea [20:55:00] ok [20:55:06] Note that it's for meta-development [20:55:26] It contains code that's run to verify the integrity of other code that's submitted by developers [20:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [21:23:44] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [21:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [21:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [21:23:54] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [21:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [21:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [21:47:04] RECOVERY dpkg-check is now: OK on driver-dev driver-dev output: All packages OK [21:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [21:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [21:53:54] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [21:54:04] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [21:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [21:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [22:15:44] PROBLEM dpkg-check is now: CRITICAL on driver-dev driver-dev output: DPKG CRITICAL dpkg reports broken packages [22:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [22:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [22:24:04] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [22:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [22:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [22:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [22:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [22:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [22:54:04] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [22:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [22:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [22:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [23:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [23:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [23:24:04] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [23:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [23:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [23:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [23:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [23:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [23:54:04] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [23:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [23:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [23:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep)