[05:59:14] 3Wikimedia Labs / 3tools: Tool Labs: Provide anonymized view of the user_properties table - 10https://bugzilla.wikimedia.org/58196#c47 (10Nemo) Active editors are not the only users changing preferences on a wiki, there are also many global users changing preferences everywhere (or in many places); only user... [07:52:33] I'm getting "ssh: connect to host tools-login.wmflabs.org port 22: Connection timed out" when I try to ssh into tools-login.wmflabs.org from bastion1. Anybody know why this is? :o [07:57:15] Well, I tried ssh FX@tools-login and it worked :D [08:51:01] 3Wikimedia Labs: Enable OpenJDK 8 - 10https://bugzilla.wikimedia.org/66171 (10Fastily) 3NEW p:3Unprio s:3enhanc a:3None I've been running my Commons bot tasks on a personal server and would like to migrate those tasks to tool labs. I updated my code to take advantage of new Java 8 features back in Mar... [09:20:17] Hi Jyothis, what are your plans concerning https://toolserver.org/~jyothis/index.html and the toolserver shutdown? [09:45:30] 3Wikimedia Labs / 3tools: Toolserver migration to Tools (tracking) - 10https://bugzilla.wikimedia.org/58788 (10merl) [09:45:31] 3Wikimedia Labs / 3tools: Provide user_slot resource in grid - 10https://bugzilla.wikimedia.org/52976 (10merl) [09:48:44] 3Wikimedia Labs / 3tools: Provide user_slot resource in grid - 10https://bugzilla.wikimedia.org/52976#c3 (10merl) I need to limit the number of some unrelated sge jobs running in paralell. For example if too much scripts editing in mediawiki are running i could hit a ratelimit (technical limited or per local... [10:12:59] 3Wikimedia Labs / 3tools: Create views for user_daily_contribs table - 10https://bugzilla.wikimedia.org/61300#c7 (10Sean Pringle) Table is replicated. [10:13:14] 3Wikimedia Labs / 3tools: Tool Labs: Provide filtered view of user_properties table containing short list of properties, linked to userID - 10https://bugzilla.wikimedia.org/64115#c3 (10Sean Pringle) Table is replicated. [10:13:15] 3Wikimedia Labs / 3tools: Create views for user_daily_contribs table - 10https://bugzilla.wikimedia.org/61300 (10Sean Pringle) a:5Sean Pringle>3Marc A. Pelletier [10:13:30] 3Wikimedia Labs / 3tools: Tool Labs: Provide filtered view of user_properties table containing short list of properties, linked to userID - 10https://bugzilla.wikimedia.org/64115 (10Sean Pringle) a:5Sean Pringle>3Marc A. Pelletier [10:13:44] 3Wikimedia Labs / 3tools: Provide filearchive table with fa_storage_key or, if it exists and is sufficiently indexed and populated, fa_sha1 for commonswiki - 10https://bugzilla.wikimedia.org/57697#c6 (10Sean Pringle) Table is replicated. [10:13:45] 3Wikimedia Labs / 3tools: Provide filearchive table with fa_storage_key or, if it exists and is sufficiently indexed and populated, fa_sha1 for commonswiki - 10https://bugzilla.wikimedia.org/57697 (10Sean Pringle) a:5Sean Pringle>3Marc A. Pelletier [12:38:28] Silke_WMDE: need to take a look - too much going on around and havent had a chance. [13:07:24] hashar: it looks to me like deployment-elastic01 doesn't run puppet? [13:07:58] manybubbles: maybe it is not pointing to the beta cluster puppetmaster? [13:08:20] I have something committed to the production branch of puppet that isn't there. I tried running sudo puppetd -tv and it complained [13:08:27] err: Could not parse /etc/puppet/puppet.conf: Could not match line at /etc/puppet/puppet.conf: [13:08:48] nice [13:09:03] iirc when we generate puppet.conf it curl some URL from the puppetmaster [13:09:18] it might have received a 404 or 500 and used the body response in the puppet.conf [13:09:20] so you might have: [13:09:45] some setting = Error not found404 The document could not be found [13:09:48] or something like that [13:10:51] certname = [13:10:51] [13:10:52] 500 Internal Server Error [13:10:52] :( [13:10:53] certname = [13:11:07]

500 Internal Server Error

[13:11:11] I am sure I mentioned that earlier and it got fixed up by ops [13:11:34] so I shoould bother an ops? [13:12:24] default => "${::ec2id}.${::domain}" [13:12:36] the ec2id is generated by a fact modules/base/lib/facter/ec2id.rb [13:14:31] manybubbles: I guess yes. Coren probably [13:14:38] the fix was https://gerrit.wikimedia.org/r/#/c/123168/5/modules/base/lib/facter/ec2id.rb,unified [13:14:52] and bug is https://bugzilla.wikimedia.org/63322 [13:15:26] manybubbles: I will try replacing the value manually [13:16:34] hashar manybubbles: Yeah, I've seen this before; facts with poor/no error checking and screw puppet up, and fixing the value manually in the manifest is the only way out. [13:16:44] putting certname = i-00000289.eqiad.wmflabs [13:16:45] [13:16:59] hashar: thanks! [13:17:10] I imagine those machines are really behind on the puppets then [13:17:16] the puppet.conf file got last changed March 31st which is when I noticed the issue and before Coren fixed it up [13:17:25] so I guess I forgot to check/verify all beta cluster instances bla [13:17:37] yeah the are behind since March 31 or so [13:17:48] manybubbles: can you check the others? [13:17:57] sure [13:18:09] adjust the certname in /etc/puppet/puppet.conf and /etc/puppet/puppet.conf.d/10-self.conf [13:18:16] and you can the id with: http://169.254.169.254/1.0/meta-data/instance-id [13:18:23] the certname need a .eqiad.wmflabs suffix [13:19:11] - Facter::Util::Resolution.exec("curl http://169.254.169.254/1.0/meta-data/instance-id 2> /dev/null").chomp [13:19:11] + Facter::Util::Resolution.exec("curl -f http://169.254.169.254/1.0/meta-data/instance-id 2> /dev/null").ch [13:19:12] \O/ [13:19:48] 03 needs it - doing [13:19:52] !log deployment-prep deployment-elastic01 puppet was broken due to {{bug|63322}} i.e. having some HTML garbage as ec2id which would be used as puppet certname [13:20:02] manybubbles: \O/ [13:20:12] needs to run puppet a few time [13:20:18] plus probably apt-get upgrade --some-magic [13:21:04] also since we migrated to an independent puppetmaster, it is hard to figure which instance has a broken puppet :-( [13:22:45] notice: /Stage[main]/Ganglia_new::Monitor::Service/Service[ganglia-monitor]/ensure: ensure changed 'stopped' to 'running' [13:22:45] pfff [13:23:12] gotta kill gmond and restart it [13:24:04] 3Wikimedia Labs / 3Infrastructure: set apt to Wikimedia mirror instead of http://nova.clouds.archive.ubuntu.com/ubuntu/ - 10https://bugzilla.wikimedia.org/66121 (10Sam Reed (reedy)) s:5normal>3enhanc [13:25:27] !log deployment-prep deployment-elastic03 also had broken puppet for the same reason - fixing [13:25:37] kudos [13:26:02] manybubbles: there is also bunch of packages pending upgrade [13:26:16] though I don't really touch elastic boxes, I am too afraid to break them [13:26:26] hashar: I'll do the updates then [13:26:49] its cool - for the most part its safe. Just only hit one Elasticsearch box at a time and it'll recover itself [13:27:01] I fixed up ganglia-monitor being relaunched every time [13:27:08] it is cause the pid file has an old pid [13:27:20] and thus puppet attempt to start while it is already starting hehe [13:27:22] started [13:27:23] gg [13:28:20] thanks [13:28:59] new kernel - I imagine I should reboot the machine too [13:29:06] I guess :) [13:30:02] !log deployment-prep rolling restart of elasticsearch boxes to get a new kernel [13:32:50] !log meplease [13:32:53] pff [13:32:56] log is dead [13:33:45] yeah, labs-morebots seems gone :/ [13:34:20] silly bot [13:47:11] manybubbles: try now? [13:47:38] !log deployment-prep rolling restart of elasticsearch nodes in beta to pick up new kernel [13:47:40] Logged the message, Master [13:50:03] andrewbogott: thanks! [13:59:23] !log deployment-prep deployment-elastic01 puppet was broken due to {{bug|63322}} i.e. having some HTML garbage as ec2id which would be used as puppet certname [13:59:25] Logged the message, Master [13:59:33] thanks [14:01:32] !log deployment-prep elasticsearch cluster got really angry in beta when I restarted some node - its like they aren't talking to eachother properly - trying to recover. once that is done I'll upgrade to 1.2.1 and that might fix it [14:01:34] Logged the message, Master [14:02:46] manybubbles: ah: Caused by: java.lang.ClassNotFoundException: org.elasticsearch.index.analysis.kuromoji.KuromojiAnalyzerProvider [14:03:06] I know why this is busted! Its because I wanted to upgrade to 1.2.1 and have already deployed new plugins.... [14:14:32] !log deployment-prep recovered from busted elasticsearch - two problems: 1. I had an index that used the kuromoji plugin but I'd uninstalled it and 2. I had plugins for 1.2.1 but was trying to start 1.1.0. Solution: 1. delete the index and recreate it without kuromoji. 2. upgrade to 1.2.1 like I had planned on doing any way. [14:14:35] Logged the message, Master [14:16:32] !log deployment-prep rebuild beta's jawiki's search index without kuromoji - it didn't help much anyway [14:16:33] Logged the message, Master [14:17:04] prtksxna: do you want to try rebuilding chicken? I suggest killing chicken and creating beef [14:17:06] prtksxna: or mutton [14:17:18] YuviPanda: Sure. Docs/ [14:18:04] prtksxna: https://wikitech.wikimedia.org/wiki/Help:Contents is the contents page [14:18:14] prtksxna: create new instance docs at https://wikitech.wikimedia.org/wiki/Help:Instances [14:18:28] prtksxna: then labs-vagrant docs at https://wikitech.wikimedia.org/wiki/Labs-vagrant [14:18:34] prtksxna: poke me at any point if you feel unsure. [14:18:38] prtksxna: also I highly suggest mutton [14:19:18] YuviPanda: Reading now. Paneer/ [14:19:35] prtksxna: no. [14:19:39] Or your favorite, boiled-egg? :P [14:20:04] prtksxna: mutton or boiledegg, yeah [14:29:28] YuviPanda: should I delete dptypes? [14:29:35] prtksxna: ya [14:29:46] prtksxna: let's go with boiledegg [14:29:55] I was thinking mutton now :P [14:30:27] YuviPanda: small would be fine? [14:30:35] prtksxna: yeah. and boiledegg! [14:32:09] YuviPanda: No changes to the Puppet stuff? [14:32:19] prtksxna: no [14:36:45] YuviPanda: I created it but I don't have SSH access on it :\ [14:36:55] prtksxna: wait 5 mins for it to boot up first [14:40:51] YuviPanda: Let's me SSH now. Can't find the mediawiki folder though… [14:42:04] prtksxna: did you enable the labs-vagrant role? [14:43:27] YuviPanda: http://pastie.org/pastes/9261222/text [14:43:42] prtksxna: do sudo puppetd -tv [14:43:43] first [14:43:47] prtksxna: and see what it says [14:45:35] YuviPanda: yup, ran clean http://pastie.org/pastes/9261229/text [14:46:14] prtksxna: checking [14:47:20] prtksxna: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=b800e152-275e-4924-b7d6-668b5db6204d&project=design®ion=eqiad [14:47:24] prtksxna: you haven't ticked the labsvagrant role [14:47:29] prtksxna: and hit save [14:48:34] [20:02:08] YuviPanda: No changes to the Puppet stuff? << That is what I was trying to ask you :P [14:48:49] prtksxna: :P labs-vagrant page told you to, so I assumed you meant any changes apart from that [14:48:51] YuviPanda: Should i reboot? [14:49:06] prtksxna: after ticking run sudo puppetd -tv [14:49:27] YuviPanda: oh, sorry! [14:51:25] YuviPanda: done [14:51:33] prtksxna: works? [14:55:37] YuviPanda: Should this have installed mediawiki too? [14:58:12] prtksxna: if you do labs-vagrant provision [14:59:03] YuviPanda: What is the difference between vagrant and labs-vagrant? [14:59:29] prtksxna: vagrant runs on your local computer, uses a VM on your computer to setup stuff [14:59:41] prtksxna: labs-vagrant runs on labs directly, and sets up stuff on the running host itself [14:59:55] prtksxna: so vagrant sets up things on a VM inside the machine where it is run, and labs-vagrant sets it up on the host itself [15:00:25] YuviPanda: But how does it matter where its running? Why isn't it simple called vagrant? [15:00:52] prtksxna: because vagrant is http://www.vagrantup.com/, and its primary purpose is to handle Virtual Machines [15:01:08] And l-v isn't on a VM? [15:01:26] prtksxna: it is a VM, but it isn't creating a VM inside a VM [15:02:41] YuviPanda: Where can I find a longer explaination? [15:03:13] prtksxna: nowhere much, since I wrote labs-vagrant :D I can do a hangout later if you want [15:03:32] YuviPanda: on 3g, will hang after I get a real connection [15:03:34] YuviPanda: oh [15:04:00] prtksxna: ok [15:04:18] one does not simple write labs-vagrant without documenting it [15:04:38] prtksxna: there is documentation! :D [15:04:43] prtksxna: it's the page I linked you to [15:28:10] YuviPanda: provisioned, /vagrant/mediawiki is empty [15:28:27] prtksxna: enable the nfs_client role? [15:29:52] YuviPanda: I am getting an empty list for the available roles (suddenly) [15:29:57] prtksxna: oh [15:30:00] prtksxna: hard refresh? [15:30:09] YuviPanda: reboot? [15:30:15] prtksxna: no, the page I mean [15:30:59] YuviPanda: check nfs::server too? [15:31:07] prtksxna: noooo [15:31:08] :D [15:31:09] just the client [15:31:31] YuviPanda: and then sudo puppetd -tv again? [15:32:36] prtksxna: yeah, and then restart again (NFS needs) [15:32:57] YuviPanda: doing… [15:33:43] * prtksxna reboots [15:50:39] \o/ [15:52:40] !log swift test log showing godog how this works [15:52:42] Logged the message, dummy [15:53:12] nice! thanks andrewbogott [15:53:36] so we are dummies to morebots' eyes here [16:11:47] YuviPanda|zzzz: Is http://boiledegg.wmflabs.org/ working for you? [18:46:44] 3Wikimedia Labs / 3Infrastructure: l10nupdate gid should be 10002 to match production/Puppet - 10https://bugzilla.wikimedia.org/65588#c3 (10Bryan Davis) 5NEW>3RESO/FIX LDAP and file system fixed. Thanks Andrew! [18:53:44] 3Wikimedia Labs / 3deployment-prep (beta): mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - 10https://bugzilla.wikimedia.org/65591#c2 (10Bryan Davis) Adding Andrew Bogott as a cc. I think the best solution to attempt for this would be to change the shell of mwdeploy in lab... [20:27:30] Coren, are there docs about how to access the db replicas from a non-tools labs project? [20:27:59] I think so. Lemme see... [20:28:47] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Connecting_to_the_database_replicas_from_other_Labs_instances [20:29:08] Though it would be sane to make a puppet class out of this for easier management. [20:29:45] Danny_B: Does that help? ^^ [20:55:50] I think http://en.wikipedia.beta.wmflabs.org/ just went down. Mobile beta seems to be still answers http://en.m.wikipedia.beta.wmflabs.org/ [20:56:04] answering, even [20:59:35] Coren: ^^ [21:00:00] nick chrismcmahonbrb [21:00:25] I'm like, the second worst person to ask. Have you checked with hashar if they're currently working on it? [21:07:26] hi, could someone help with creating a new instance in beta cluster? [21:07:54] we could really use it for zerowiki [21:17:08] yurikR: you are an admin in the project, which means you should be able to create instances yourself. [21:17:27] andrewbogott, i guess i am looking for a step-by-step :) [21:17:53] than again, here i go... hope i don't break anything... btw, is beta cluster down? [21:17:56] yurikR: like this? https://wikitech.wikimedia.org/wiki/Help:Instances [21:18:27] andrewbogott, i thought beta cluster has a different procedure [21:18:36] and its not the instance, its the wiki i need [21:18:37] It may, but if so... [21:18:47] well, if so, I'd advise asking one of the beta admins (you, for example :/ ) [21:19:04] all right all right, just didn't want to break anything ) [21:20:57] Coren, on Monday you mentioned wanting some time from me to fix some bugs… want to send me a laundry list? [21:21:14] (Note that most wikitech bugs are blocked for want of a test instance, which I keep trying to work on but getting pulled away...) [21:21:39] andrewbogott: I got distracted this week by personal gunk; I'll try to sit down with you tomorrow? [21:22:02] Coren: Sure. (Although much of tomorrow will be devoted to peer reviews, I expect) [21:22:13] andrewbogott: I done did mine. :-) [21:22:27] is it possible to change the configuration and/or name of the instance? [21:23:25] Danny_B: name of instance, no. But, what do you mean by 'configuration'? [21:23:43] the model or however it is called [21:23:55] where you set the # of cpus, ram and hdd [21:24:08] Ah, nope, that's all fixed on creation time. [21:25:38] so basically when i need bigger with same name (foo), i need to create foonew, copy stuff from foo to foonew, delete foo, create bigger foo, move foonew to foo, delete foonew, right? [21:27:18] um… essentially. Or you can just move files to /data/project [21:27:21] which is shared among instances. [21:27:26] Then you could just delete foo and recreate it [21:29:07] ah, makes better sense [21:29:47] Also, the ssh host key changes if you recreate an instance with the same name. [21:29:51] btw are you guys aware of instance creation (and later configuration) throws bunch of errors? [21:30:30] throws them where? [21:30:36] console log [21:54:44] 3Wikimedia Labs / 3deployment-prep (beta): shell wrapper to connect to databases - 10https://bugzilla.wikimedia.org/45706#c4 (10Yuri Astrakhan) This also works: mwscript sql.php --wiki=aawiki [22:48:11] hello hello [22:50:02] a year ago (!) I had requested a new git repo for the LoopFunctions extension, and ^demon created it, but I never took the time to try it until today [22:50:07] now it seems that I can't push [22:50:34] the message on https://www.mediawiki.org/wiki/Git/New_repositories/Requests/Archive is "Done Repository & group created, user does not seem to be in Gerrit. Please let me know when you have a Gerrit account. ^demon[omg plz] 22:12, 10 May 2013 (UTC)" [22:51:00] we use a code review system, we don't usually do direct pushes to repositories [22:51:15] I don't know why I didn't appear in gerrit, but I put my key in it and it seems to work [22:51:38] ori: so how am I supposed to publish my commits in my repo? [22:51:40] are you trying to import an existing repository, or is this a new project? [22:51:50] no, it's an existing project [22:52:15] it was maintained in the page on wikimedia.org, and I then created a private git repo [22:52:37] today I cloned the gerrit repo and imported my commits, tags, etc. in it [22:52:52] does that make sense? [22:53:16] s/wikimedia.org/mediawiki.org/ [22:53:27] it's there: https://www.mediawiki.org/w/index.php?title=Extension:LoopFunctions&oldid=209910 [22:53:33] oops, I'm tired [22:53:37] https://www.mediawiki.org/wiki/Extension:LoopFunctions [22:54:39] and the message I get with git push: ! [remote rejected] REL1_23 -> REL1_23 (prohibited by Gerrit) [22:54:52] (I guess this is normal, if I understand correctly, but just checking) [22:56:05] ori: any idea how I'm supposed to do? [22:57:33] i'll grant you push rights [22:58:05] it's a tiny extension which I didn't initially write but ended up maintaining for my own needs, and I'm by no mean a MW or PHP developper, so code review would be great, but I would like to publish the last version ASAP [22:58:09] (because I wrote it a long time ago and forgot to publish it, and now I see people had trouble with newer MW) [22:58:12] ori: great [22:58:46] so I'll be able to push, but for later commits I should go through code review instead of pushing directly, correct? [22:59:11] Naha: what is your gerrit username again? [22:59:21] ori: Xiloynaha [22:59:30] (and naha for shell access) [22:59:35] Naha: yes [22:59:53] in fact, i'll ask you to push, and then i'll remove the push right so you can go through gerrit for additional changes [23:00:01] if that's ok [23:00:07] that's perfect, thank you [23:00:37] I'll have to read the doc, I saw several long pages -_- [23:00:57] Naha: could you push now? [23:01:07] yep [23:01:13] can I push tags as well? [23:01:39] yes [23:02:25] ok it's all good [23:02:33] except the tags don't appear on https://git.wikimedia.org/summary/mediawiki%2Fextensions%2FLoopFunctions [23:03:23] i see the tags [23:04:06] ori: where? [23:04:37] Naha: http://i.imgur.com/7AIObnl.png [23:05:13] browser cache? [23:05:20] yeah :-] [23:05:25] thanks [23:05:59] np [23:06:03] ori: you can remove the push right now [23:06:12] thanks for your help [23:06:31] and I'll go read the doc next time I want to publish a commit I guess [23:08:01] Naha: yes, try not to get discouraged by how awful gerrit is [23:08:14] we'll be replacing it with a nicer system (phabricator) soon [23:08:28] Naha: if you despair, ping me and i'll grant you direct push rights again [23:08:42] but be aware that this will make it more difficult to have your extension deployed on Wikimedia's cluster [23:12:04] ori: I patch it once every two years when it's broken by a new MW version anyway, so next time the new system will probably be in place :-) [23:12:26] ori: how is Gerrit so awful? [23:15:55] I'm reading https://wikitech.wikimedia.org/wiki/Help:Git, it sounds like it's as simple as pushing with git review [23:16:59] Naha: yep [23:19:59] so, can you tell me more about the badness of gerrit, and how phabricator is better? (if you have time) [23:20:48] Naha: don't really have the time, and i should actually not bias you against gerrit if i can help it. it's perfectly okay to like it if it works for you :) [23:20:59] Naha: i find the UI a bit off-putting, but it's subjective in the end [23:21:15] hehe OK, I see [23:22:01] one last question, do I have a mean to request a code review on the whole extension, now that it is in the repo? [23:22:31] you can file a bug in bugzilla requesting it [23:22:52] or ask on wikitech-l (https://lists.wikimedia.org/mailman/listinfo/wikitech-l) [23:23:07] OK thanks, I'll see that