[00:12:05] Coren, when I go to https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000002a1.eqiad.wmflabs and click 'configure', I get. "The specified resource does not exist." [00:33:45] 3Wikimedia Labs / 3deployment-prep (beta): "404 file Not Found Error" when logging into betalabs - 10https://bugzilla.wikimedia.org/71806 (10Greg Grossmeier) a:3Sam Reed (reedy) [00:47:24] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools.diskspace._var.byte_avail.value (11.11%) [01:04:10] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [01:53:43] 3Wikimedia Labs / 3deployment-prep (beta): "404 file Not Found Error" when logging into betalabs - 10https://bugzilla.wikimedia.org/71806#c7 (10Sam Reed (reedy)) 5NEW>3RESO/FIX Another example of why beta shouldn't have a diverged apache config from production. Docroot paths fixed in https://gerrit.wiki... [02:33:57] superm401: it works for me. Most likely you're hitting the session bug and need to log out and back in. [02:34:14] We merged a patch earlier today that will fix that problem, I hope. [02:38:15] Yeah, that fixed it. [05:12:30] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c6 (10Greg Grossmeier) a:3Sam Reed (reedy) Reedy: Another docroot issue? This and bug 70948, too. [06:08:01] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862 (10Matthew Flaschen) 3NEW p:3Unprio s:3major a:3None On Beta Labs enwiki, attempting to create an account gives (in the red error box): --- Ac... [06:08:13] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c1 (10Greg Grossmeier) p:5Unprio>3Normal Matt: Do you have access to the Beta Cluster? If not, you should. Then can you poke around the logs and se... [06:08:29] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c2 (10Matthew Flaschen) I think I have the necessary access. I'm not planning to take this (at least not tonight), though. [06:08:43] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c3 (10Matthew Flaschen) S Page said he was able to create an account, but it then redirected to the a page that said, "You are already logged in as Spa... [06:09:15] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c4 (10Greg Grossmeier) Mini rant: Those with production deploy privs should have access to the beta cluster (if not, I'll add you right now) and should... [06:10:44] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c5 (10Greg Grossmeier) (not complaining about you two specifically, Matthew/S, just something that's been on my mind for a while about making our test... [06:30:12] http://en.wikipedia.beta.wmflabs.org/w/api.php 503? [06:47:50] PROBLEM - ToolLabs: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: tools.tools-exec-09.puppetagent.failed_events.value (30.00%) [07:13:10] RECOVERY - ToolLabs: Puppet failure events on labmon1001 is OK: OK: All targets OK [09:03:04] Hello; My tool depends on a Python module (pygal) that's not available in tool labs. What's the recommended way of installing it? [09:03:43] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783#c6 (10Antoine "hashar" Musso) Creating an instance deployment-cxserver02 : Size: m1.medium OS: Ubuntu Trusty Security rules: default, cxserver Ie the same as deploym... [09:09:20] guillom: good morning :-] [09:09:26] salut hashar :) [09:09:59] the module should be available as a Ubuntu .deb package [09:10:15] then added to operations/puppet.git in modules/toollabs/manifests_exec_environ.pp [09:10:27] at least that is where a looooot of python-* packages are made available [09:10:35] then puppet will ensure the module is installed everywhere [09:10:52] hashar: looks like it's only packaged for Fedora and Gentoo: http://pygal.org/download/ ; on my local machine I simply did a "pip install pygal". [09:10:56] :-( [09:12:07] guillom: I found a request for packaging it https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741653 :( [09:13:23] guillom: I am not sure whether they allow installing packages via pip (puppet has support to do so) [09:13:36] might want to bug fill it under Wikimedia Labs > tools [09:14:16] and it needs a few more python packages : pip install CairoSVG tinycss cssselect :-( [09:14:44] hashar: I might just download it locally then [09:14:46] as a workaround, and assuming none are compiled packages, you could ship the dependencies along your project [09:15:10] pip install --user might do it [09:15:33] though I think tools can roam between instances so I am not sure how it will work ( your /home is probably available on all instances anyway) [09:18:43] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783#c7 (10Antoine "hashar" Musso) Kart confirmed we can get rid of the instance. Since beta cluster is out of quota, that is convenient. [09:19:15] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783#c8 (10Nemo) (Context: > The virt1005 compute node died overnight, might explain the issue. https://lists.wikimedia.org/pipermail/labs-l/2014-October/002982.html ) [09:39:20] guillom: hey! [09:39:30] guillom: if you can install it with pip, you can just use a virtualenv and install it there [09:39:36] YuviPanda: hello :) [09:39:53] YuviPanda: I... have no idea how to do that :) [09:40:10] guillom: ah, I can help! [09:40:18] I'd be very grateful! [09:40:29] guillom: does this tool have a web interface, or is it just a bot? [09:41:05] YuviPanda: it has a web interface ( https://tools.wmflabs.org/mrmetadata/index.html ) [09:41:39] though it's just static HTML files generated by the script [09:41:49] guillom: ah, I see. [09:41:58] guillom: so your python code just statically generates the files. [09:42:04] yep [09:42:14] guillom: cool. [09:42:26] guillom: so, in your tool directory, run 'virtualenv .' [09:42:36] ok [09:42:58] installing [09:42:59] done [09:43:08] guillom: cool. now do 'source bin/activate' [09:43:23] YuviPanda: done [09:43:41] guillom: cool, now you can just do 'pip install ' and it will work :) [09:43:56] guillom: what we have done is created an isolated virtual environment in your tool's home directory, with its own set of python libraries. [09:44:11] currently it has no python libraries, so you might have to pip install all the ones you need. [09:44:17] YuviPanda: oh, cool! [09:44:44] YuviPanda: Oh, so I need to install the usual suspects [09:44:47] guillom: yup [09:44:50] ok [09:44:52] guillom: it's fully isolated. [09:45:03] guillom: also, in your python script, I see a hashbang of /usr/bin/python [09:45:13] guillom: you will want to change that to /data/project/mrmetadata/bin/python [09:45:20] got it [09:45:23] guillom: since we want to use our isolated environment's python + its libraries [09:45:30] YuviPanda: anything else I need to do differently to run the tool? [09:45:52] guillom: nope! Just make sure that the python executable used is the one in your tool's bin folder, and it should be all good :) [09:46:04] YuviPanda: like, in the jsub command, use the virtualenv python path? [09:46:08] guillom: yup [09:46:12] okay [09:46:30] YuviPanda: I'll try and I'll ask again if I get stuck. Thank you so much :) [09:46:35] guillom: another reccomendation is, after you've done installing all the modules you need, run 'pip freeze > requirements.txt' [09:46:45] guillom: that will output all the libraries + exact versions you are using [09:47:00] ok, thanks! [09:47:09] guillom: and then you can save that .txt file (commit it to git perhaps), and if you want to re-create the exact same libraries elsewhere, you can just do 'pip install -r requirements.txt' [09:47:18] thus all the dependencies of your script are neatly documented [09:47:20] guillom: yw! :) [09:47:25] ahah! cool [09:48:06] guillom: virtualenv is cool :) Highly reccomended, both for running apps and for development on your machine locally :) [09:48:35] YuviPanda: as you might have guessed, I'm new to Python. Thank you for the help! [09:48:51] guillom: indeed, happy to see you using it! it's a really nice language :) [09:49:02] It is indeed. [09:49:14] guillom: do poke me if I could help anymore. I find #python rather friendly as well. [09:49:22] thanks! [09:49:39] yw [09:49:44] * YuviPanda goes off to hunt for some food [10:03:29] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783 (10Antoine "hashar" Musso) [10:03:31] 3Wikimedia Labs / 3Infrastructure: role::labs::lvm::mnt ends up with make-instance-vg: failed to create new partition - 10https://bugzilla.wikimedia.org/71873 (10Antoine "hashar" Musso) 3NEW p:3Unprio s:3normal a:3None On beta cluster we are making use of role::labs::lvm::mnt to allocate the instance... [10:12:00] 3Wikimedia Labs: Unable to mount /mnt on integration-slave1009 - 10https://bugzilla.wikimedia.org/71874 (10Krinkle) 3NEW p:3Unprio s:3normal a:3None -- Puppet run log -- Info: Caching catalog for i-00000671.eqiad.wmflabs Info: Applying configuration version '1412716544' Notice: /Stage[first]/Apt::Updat... [10:12:59] 3Wikimedia Labs / 3Infrastructure: WMFLabs: New instances with precise image are broken (puppet run fails, no ssh access possible) - 10https://bugzilla.wikimedia.org/71741#c4 (10Krinkle) I don't think the ldap thing is the problem. The log I pasted in comment 0 shows that it tried both. It's failing for a di... [10:19:31] 3Wikimedia Labs / 3Infrastructure: role::labs::lvm::mnt ends up with make-instance-vg: failed to create new partition - 10https://bugzilla.wikimedia.org/71873#c1 (10Krinkle) *** Bug 71874 has been marked as a duplicate of this bug. *** [10:19:31] 3Wikimedia Labs: Unable to mount /mnt on integration-slave1009 - 10https://bugzilla.wikimedia.org/71874#c1 (10Krinkle) 5NEW>3RESO/DUP *** This bug has been marked as a duplicate of bug 71873 *** [10:20:14] 3Wikimedia Labs / 3Infrastructure: role::labs::lvm::mnt ends up with make-instance-vg: failed to create new partition - 10https://bugzilla.wikimedia.org/71873#c2 (10Krinkle) p:5Unprio>3Normal s:5normal>3critic The same thing has been happening since last week on integration-slave1009. I never got to... [15:07:28] 3Wikimedia Labs / 3Infrastructure: WMFLabs: New instances with precise image are broken (puppet run fails, no ssh access possible) - 10https://bugzilla.wikimedia.org/71741#c5 (10Andrew Bogott) I just created new images last night which seem generally happier. Try again? [15:28:51] andrewbogott: the wikitech vagrant role was borked, just merged a fix. [15:34:14] !paste [15:34:14] http://tools.wmflabs.org/paste/ [15:35:25] There [15:35:37] There's a paste in the new phabricator too [15:50:41] <^d> bd808: Hmm, is there anything in deployment-prep we can drop? [15:50:51] <^d> Running too close to our quotas to create new elastic box. [15:51:01] bah. Ask for moar! [15:51:33] Or tell me where the page to give us more is. I have godlike wikitech powers now [15:51:47] bd808: it requires shell on virt*, I think [15:51:50] not on wikitech [15:51:54] boo [15:52:04] * bd808 looks at instance list [15:52:08] <^d> It's the number of instances we're hitting, 43/43. [15:52:20] <^d> Still have enough cores/ram/ [15:52:46] <^d> deployment-lucid-salt [15:52:49] <^d> lucid wut? [15:53:24] <^d> Oh, I can kill deployment-saio [15:53:56] that salt server is apergos testing the salt upgrade [15:54:03] It will die soon [15:54:39] The pdf and sca boxes could probably be consolidated [15:54:46] but I don't want to do it :( [15:55:27] And rsync and udplog could be combined I bet [15:56:06] Or we could just ask for more quota. Seems like sniping at instances is a waste of time. [15:56:17] bd808: you should just bug andrewbogott for more quota [16:02:31] ^d, YuviPanda: https://bugzilla.wikimedia.org/show_bug.cgi?id=71886 [16:02:48] Feel free to suggest moar than I asked for [16:08:55] andrewbogott: I can't really test anything that uses openstack manager locally, can I? [16:09:38] You can use the wikitech-test-* cluster in labs [16:09:40] e.g. wikitech-test-frontend [16:09:47] andrewbogott: does that access the openstack api? [16:09:58] yes, there's an openstack install running there [16:10:05] oh wow [16:10:08] openstack on openstack [16:10:08] nice [16:10:17] will do! [16:10:22] andrewbogott: thanks! [16:15:01] 3Wikimedia Labs: Increase quota for deployment-prep (beta) project - 10https://bugzilla.wikimedia.org/71886 (10Bryan Davis) 3NEW p:3Unprio s:3normal a:3None We are bumping our head against the instance limit again. Chad had to find an instance to kill so he could start migrating the cirrus servers to 1... [16:45:45] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783 (10Greg Grossmeier) p:5Unprio>3Normal [16:45:45] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783#c9 (10Greg Grossmeier) CRITICAL: deployment-prep.deployment-cxserver02.puppetagent.failed_events.value (100.00%) [16:48:24] role::puppet::self is the one to be ticked to make the instance a puppetmaster right ? [16:57:12] tonythomas: so far so good. just need to work out a couple of glitches [16:57:41] first I need to figure out how to make git ignore the locally modified manifests/site.pp [17:02:39] Jeff_Green: the next time we git pull right ? [17:03:00] I dont think that would happen automatically, once we are having our own puppetmaster [17:04:06] i added beta::puppetmaster::sync which does a rebase hourly on cron [17:04:24] and I added a second cron job to make that every 10 min :-P [17:04:32] Jeff_Green: oh. in that case, I think we should get that out of git [17:04:35] horrible horrible hacking. I feel dirty. [17:04:58] so far so good, it's failing on missing dkim keys in the private repo [17:05:12] Jeff_Green: that can be ignored I think ? [17:06:13] i think we need keys to test outbound mail, but we should be able to work with dummy keys [17:06:28] ok. the puppet run finished ? [17:06:35] yeah. [17:07:24] tonythomas: you have root access on this instance? [17:07:32] nope :\ [17:07:37] :-( [17:07:41] I dont have apparently any access in that instance [17:07:44] i was going to invite you to make keys :-P [17:07:55] oh great. [17:08:12] but it looks like, I'm locked out [17:08:20] anyway - the puppet is still running ? [17:08:31] * tonythomas feels like watching the soccer game [17:10:08] no it's done [17:10:56] if you want to go watch the game, I've got fundraising stuff I should get done, we can pick up later? [17:18:21] Jeff_Green: great [17:18:46] anyway, I was referring to the game over here in #labs :D anyway, catch ya after your work [17:51:25] good morning [17:51:56] I'm looking for a way to update an extension in beta labs [17:52:09] but don't even see the host list in wikitech right now [17:52:39] the automatic update seems to have a race condition & deployed an earlier version [17:53:10] gwicke, go to deployment-bastion and scap? [17:55:01] deployment-bastion is laid out like prod. The extensions are updated with the https://integration.wikimedia.org/ci/view/Beta/job/beta-code-update-eqiad/ Jenkins job [17:55:59] It pulls all extensions from the mediawiki/extensions repo [17:56:19] And will hard reset to that state each time the job runs [18:00:02] hmm, no andrewbogott [18:01:06] MaxSem helped me, currently waiting for the scap lock file [18:02:00] the checkout on bastion was up to date, but the Jenkins job showed an earlier version & the deployed code seems be the older version too [18:02:34] is there a full scap at regular intervals? [18:03:06] gwicke: Yes, after each beta-code-update-eqiad job runs [18:03:29] gwicke: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ [18:06:02] k [18:16:02] 3Wikimedia Labs / 3deployment-prep (beta): Requested 115.108.187.192.proxies.dnsbl.sorbs.net., not found in proxies.dnsbl.sorbs.net.. - 10https://bugzilla.wikimedia.org/71894 (10Sam Reed (reedy)) 3NEW p:3Unprio s:3minor a:3None Requested 115.108.187.192.proxies.dnsbl.sorbs.net., not found in proxies.... [18:16:26] I will let you know when I see andrewbogott around here [18:16:26] @notify andrewbogott [18:46:28] 3Wikimedia Labs / 3deployment-prep (beta): Requested 115.108.187.192.proxies.dnsbl.sorbs.net., not found in proxies.dnsbl.sorbs.net.. - 10https://bugzilla.wikimedia.org/71894#c1 (10Yuvi Panda) Where's this coming from? [18:47:29] 3Wikimedia Labs / 3deployment-prep (beta): Requested 115.108.187.192.proxies.dnsbl.sorbs.net., not found in proxies.dnsbl.sorbs.net.. - 10https://bugzilla.wikimedia.org/71894#c2 (10Sam Reed (reedy)) I noticed it in logstash-beta [18:49:43] 3Wikimedia Labs / 3deployment-prep (beta): Requested 115.108.187.192.proxies.dnsbl.sorbs.net., not found in proxies.dnsbl.sorbs.net.. - 10https://bugzilla.wikimedia.org/71894#c3 (10Sam Reed (reedy)) { "_index": "logstash-2014.10.09", "_type": "dnsblacklist", "_id": "CeX-WiVqTPerC4N8uRXoog", "_score":... [18:50:17] hey andrewbogott [18:50:24] * andrewbogott waves [18:50:45] andrewbogott: https://gerrit.wikimedia.org/r/#/c/165755/ mind if I merge? [18:51:04] me and legoktm (mostly legoktm :P) are working on adding API to get list of projects, and then list of instances + info per each project... [18:53:32] andrewbogott: http://wikitech-test.wmflabs.org/w/api.php?action=query&list=novaprojects&format=jsonfm :) [18:54:04] ok then! [18:54:06] legoktm: I +2'd :D [18:54:15] the next one now... :) [18:55:19] YuviPanda, legoktm, remember that to get them merged onto wikitech they'll have to make it into a wmf mediawiki branch, etc. etc. [18:55:37] andrewbogott: yeah, I'll SWAT if it doesn't make it [18:56:10] andrewbogott: have you seen https://tools.wmflabs.org/nagf/?project=tools [18:56:33] so many graphs [18:58:58] andrewbogott: heh :) [18:59:01] andrewbogott: automatically generated. [19:04:58] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c6 (10Sam Reed (reedy)) Can someone test this for me now? [19:15:28] 3Wikimedia Labs / 3deployment-prep (beta): Requested 115.108.187.192.proxies.dnsbl.sorbs.net., not found in proxies.dnsbl.sorbs.net.. - 10https://bugzilla.wikimedia.org/71894#c4 (10Antoine "hashar" Musso) On beta we have: # Attempt to auto block users using faulty servers # See also http://www.us.sorbs.net/... [19:18:28] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783#c10 (10Antoine "hashar" Musso) (In reply to Greg Grossmeier from comment #9) > CRITICAL: > deployment-prep.deployment-cxserver02.puppetagent.failed_events.value > (100... [19:19:59] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783#c11 (10Greg Grossmeier) (In reply to Antoine "hashar" Musso from comment #10) > (In reply to Greg Grossmeier from comment #9) > > CRITICAL: > > deployment-prep.deploym... [19:22:38] andrewbogott: Coren hmm, puppet fails to run on tools-webproxy, with: [19:22:39] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to fetch instance ID at /etc/puppet/modules/base/manifests/init.pp:198 on node i-000000e6.eqiad.wmflabs [19:23:08] * YuviPanda digs [19:23:58] hmm, becauuse if($::ec2id == '') { [19:24:37] YuviPanda: I can look in a few minutes. That value should be coming from a custom fact, ec2id [19:24:44] andrewbogott: yeah [19:28:53] andrewbogott: can you give me and legoktm cloud admin on wikitech-test? [19:29:03] and the username `Baz` [19:29:15] YuviPanda: wait no I can do it [19:29:18] one minute :P [19:29:30] haha [19:29:32] ok [19:30:15] <^d> bd808: So I started trying to rebuild the elastic hosts on beta. Copied the config from elastic01. I get all busted up when I try setting up the alternative puppet/salt for deployment. [19:30:22] > Baz (Talk | contribs)‏‎ (cloudadmin, shell) (Created on 9 October 2014 at 19:15) [19:30:32] <^d> $ sudo puppet agent -t [19:30:33] <^d> Exiting; no certificate found and waitforcert is disabled [19:31:15] ^d: someone has to explicitly sign the cert on the puppet master. [19:31:19] "sudo puppet cert list" [19:31:25] and then "sudo puppet cert sign " [19:31:59] <^d> I didn't know creating a host in deployment-prep was so complicated :) [19:31:59] YuviPanda: no change [19:32:13] bah [19:32:25] legoktm: maybe you need to be part of the project? [19:32:41] ugh [19:33:06] legoktm: I've never touched that part of code before... [19:33:24] legoktm: can you read parts of it in the UI? [19:34:06] ^d: It is a complex dance. Documented by complex [19:34:10] *but [19:34:55] <^d> Ok, I'm listing the certs on deployment-salt. [19:34:57] <^d> I see two [19:35:10] There's probably a similar step needed to sign the salt cert. [19:35:27] ^d: one of them is probably the id of the instance you are trying to enable -- sign that one :) [19:35:35] legoktm: as in, the cvn project, if you're cloudadmin... [19:35:56] <^d> Ah, it is! [19:37:08] <^d> andrewbogott: Oh yay it's all fixed now [19:37:10] <^d> Thanks! [19:38:12] <^d> Notice: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: Error: Can't create any more partitions. [19:38:12] <^d> Notice: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: /usr/local/sbin/make-instance-vg: failed to create new partition [19:38:14] <^d> Ok, progress. [19:38:18] <^d> lvm on trusty [19:38:30] ^d: yup, deployment-cxserver also has same issue [19:38:37] cxserver02 [19:39:09] <^d> Any workaround or way to fix? [19:39:45] YuviPanda: okay, progress! http://wikitech-test.wmflabs.org/w/api.php?action=query&list=novainstances&niproject=cvn&format=jsonfm&niregion=eqiad [19:39:56] ah, yay? [19:40:14] legoktm: do you get same NovaUser does not exist even if logged out? [19:40:23] yes [19:40:30] YuviPanda: is deployment-cxserver a brand new instance? [19:41:05] YuviPanda: the error message is https://wikitech.wikimedia.org/wiki/MediaWiki:Openstackmanager-nonovacred-admincreate [19:41:31] andrewbogott: deployment-cxserver02? I don't know [19:41:33] let me check [19:42:04] andrewbogott: it's not, I see log files back from aug 1 [19:44:34] YuviPanda: if you know how to fix that error, I think it should work [19:46:44] legoktm: looking around... [19:48:12] legoktm: I see, that's because the ldap user does not exist [19:48:24] legoktm: which creds is this using, btw? [19:48:25] shouldn't it get created on account creation? [19:48:28] nfi [19:48:30] oh [19:48:37] username: "Baz" password: "1" [19:48:43] hmm [19:49:38] legoktm: wikitech-test contains an old duplicate of production ldap, but none of the corresponding projects have actually been set up in openstack. [19:49:53] So there will always be some out-of-sync issues for things that weren't explicitly created /on/ the wikitech-test cluster. [19:49:59] (not sure if that's related to what you're seeing) [19:50:37] andrewbogott: hmm, does creating a user in wikitech-test create one in the ldap it uses? [19:50:40] it should, I suppose... [19:50:40] andrewbogott: basically I need to login as an ldap user, I thought creating a new account would work, but it didn't [19:50:54] probably! [19:54:08] * ^d kicks trusty a few times [19:55:29] andrewbogott: are there docs on how to do ldapsearch in that test instance? [19:55:33] * YuviPanda hasn't really played with LDAP before [19:55:45] no, but -- just use -h localhost [19:56:53] andrewbogott: on wikitech-test-frontend? [19:56:58] right [19:57:32] hmm, ldap_sasl_interactive_bind_s: Unknown authentication method (-6) [19:57:54] ah, hmm [19:57:57] let me read the ldapsearch docs [19:58:19] I think you want -x as well... [19:59:21] andrewbogott: yeah, that's better [20:00:27] andrewbogott: what string should I use? dc=wikimedia,dc=org, but before that...? [20:00:42] what are you trying to search for? [20:00:56] andrewbogott: wether the user Baz exists [20:01:30] I would say ldapsearch -x -h localhost -b ou=people,dc=wikimedia,dc=org but that returns an empty list [20:02:08] yeah, I tried that first... [20:08:58] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c7 (10Matthew Flaschen) (In reply to Sam Reed (reedy) from comment #6) > Can someone test this for me now? Yeah, still the same symptoms for me. Any... [20:16:13] 3Wikimedia Labs / 3Infrastructure: Jenkins can not ssh to deployment-cxserver01 (hosted by virt1005) - 10https://bugzilla.wikimedia.org/71783#c12 (10Yuvi Panda) So puppet fails on cxserver02 because it tries to create a lvm volume and fails (/mnt, I think), leading to cascading failures (among which this is... [20:17:04] legoktm: I'm turning up empty atm, sadly... [20:17:07] :/ [20:17:32] umm [20:17:43] YuviPanda: can we just take over one of the existing accounts? [20:17:56] legoktm: yeah, but I've no way of verifying if that account exists, tho. [20:17:58] but try? [20:18:26] http://wikitech-test.wmflabs.org/wiki/Special:ListUsers [20:18:29] which one? [20:18:34] "Lemonpie" [20:18:54] legoktm: andrew bogott :P [20:19:08] if you say so! [20:19:39] <^d> manybubbles: How are the disks and mounts setup on deployment-elastic0[1-4]? It looks different from what puppet's wanting to do. [20:20:17] ^d: it's not unique to es, btw. deployment-cxserver02 also has the exact same issue [20:20:23] re: puppet and lvs [20:20:26] err [20:20:26] lvm [20:20:34] <^d> Yeah, I'm trying to figure out a way around this :) [20:20:34] YESSSSSSS [20:20:38] it's not broken! [20:20:44] it's not working, but not broken! [20:21:13] hah! [20:26:30] YuviPanda: um, do you know which projects have instances on wikitech-test? [20:26:40] andrewbogott: ^ do you? [20:26:42] I... don't [20:26:51] legoktm: can you see them in the UI? [20:26:58] oh let me do that [20:28:15] legoktm: you're cloudadmin, so should be able to see everything [20:29:59] YuviPanda: http://cl.ly/image/1R3f0A0o0x27 [20:30:57] legoktm: lol [20:30:59] any instances at all? [20:31:05] still loading [20:31:40] I'd suggest you just create a new project there. [20:31:51] Instances don't actually spin up but you can 'create' one enough to test with. [20:33:02] YuviPanda: no instances [20:33:05] so my API worked! [20:33:06] maybe [20:33:10] okay [20:33:27] will create a new project [20:33:30] haha [20:33:30] ok [20:34:44] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c8 (10Sam Reed (reedy)) Not sure straight off. Memcached was in somewhat of a mess on beta, which was spamming the hell out of the logs. I sorta presum... [20:34:57] YuviPanda: Fatal error: Maximum execution time of 30 seconds exceeded in /srv/vagrant/mediawiki/extensions/LdapAuthentication/LdapAuthentication.php on line 310 [20:35:00] think I broke it. [20:35:09] legoktm: when trying to create a new project? [20:35:13] legoktm: try creating a new instance, maybe? [20:35:19] that was trying to visit Special:NovaProjects [20:35:27] how do I create a new project? [20:35:47] legoktm: should be at the top of Special:NovaProject if you're cloudadmin [20:35:59] well I can't get the page to load! [20:36:06] my selector thingy is too big [20:36:10] ow [20:36:12] ... [20:37:30] * legoktm tries again [20:41:08] page is still loading... [20:42:00] YuviPanda: I don't see a create project button [20:42:17] oh [20:42:19] not a cloudadmin [20:42:28] legoktm: become one! :) [20:44:02] Two-factor authentication required [20:44:02] Two-factor authentication is required. Please enable it and try again. [20:44:03] wat [20:44:31] * legoktm disables that check [20:45:22] legoktm: wat, lol. I didn't know... [20:53:18] YuviPanda: it's still creating the instance [20:53:37] legoktm: it's probably dead... [20:54:25] hi, i realize this isn't the right place to ask this Q, but hoping someone can point me to the right place: i need help completing a wikimedia login unification process that's been stalled + can't complete. [20:55:25] jm3: #wikimedia-stewards [20:55:47] legoktm: thx! [20:59:49] legoktm: andrewbogott I wonder if we should say 'fuck it' and test in production.... [20:59:50] * YuviPanda hides [21:10:04] um [21:10:09] YuviPanda: can you even get http://wikitech-test.wmflabs.org/ to load? [21:10:30] legoktm: hmm, no... hangs forever... [21:10:38] Hi together! Any Ideas: https://tools.wmflabs.org/denkmalliste gives 404, changed nothing, just restarted webservice [21:10:56] YuviPanda: lemme try restarting apache [21:10:59] legoktm: restarting it now [21:11:02] oh [21:11:05] instance or ? [21:11:43] legoktm: is back up [21:11:50] legoktm: we probably overloaded memory or something [21:11:55] ok my instance is "building" [21:11:57] andrewbogott: has an instance ever been successfully created there? [21:12:10] YuviPanda: http://wikitech-test.wmflabs.org/w/api.php?action=query&list=novainstances&niproject=api-test&format=jsonfm&niregion=eqiad [21:12:28] legoktm: 'NovaUser does not exist'? [21:12:33] errr [21:12:33] is that what you're getting as well? [21:12:48] legoktm: i'm not logged in, btw [21:12:55] YuviPanda: login as Andrew Bogott, guess the password [21:12:57] ;) [21:13:15] legoktm: tellllllmeeeee [21:13:26] legoktm: not hunter2 [21:14:30] its the password I use on all test wikis :P [21:14:53] legoktm: YAAAAYYY [21:14:58] legoktm: but we can't make it work for anons? [21:15:02] because of $wgUser? [21:16:01] I don't know [21:16:05] it does some scary stuff.. [21:16:36] YuviPanda: It depends on what you mean by 'successfully'. [21:16:44] Yes, created enough to be registered in the nova db [21:16:50] andrewbogott: haha :) built and booting? [21:17:10] no, definitely not booting [21:17:15] But that shouldn't matter for what you're testing [21:18:30] no one any sugestions? :( [21:18:31] andrewbogott: did you see my masseage about etherpad instance? [21:18:40] matanya: nope! [21:18:42] YuviPanda: is this ok if it doesn't work in XML? [21:19:07] andrewbogott: safe to delete [21:19:11] andrewbogott: true. [21:19:22] legoktm: would be nice if it did, in way of not fucking up the code more... [21:19:37] matanya: great, thank you. [21:19:47] sure [21:19:58] well I can do it but blegh. [21:20:20] YuviPanda: y'all both have root on all those boxes… what's the problem? [21:21:01] andrewbogott: mostly presuming you know the code better :) but yeah, I'll stop bugging you unless I really hit a wall. [21:21:29] I got a new phone since I used that site last, so I'd have to dig around and reset 2fa. Can do that if need be. [21:21:43] no, I just disabled 2FA checking [21:21:49] andrewbogott: nah, should be ok. we've usurped your user account anyway :) [21:21:58] fair enough [21:22:49] Hey, not sure if this is the best channel to ask but I didn't get a response in wm-dev; I signed up earlier today for Phabricator (using my LDAP account) and I now tried to connect my MediaWiki account to it. However, that didn't work because my account isn't part of a global account; and I can't make it a global account as I don't own the account on the "home wiki". Is there anything I can do about this? [21:23:16] poke`: sorry to keep redirecting you, but try #wikimedia-devtools [21:23:30] Sure, thanks :D [21:24:10] legoktm: I don't think requiring login is too much of an issue, but I'm wondering why we can't just use the nova creds and login for the user... [21:24:50] YuviPanda: maybe in a follow-up patch sure. [21:25:02] legoktm: hmm, cool. get xml working and I'll merge? [21:25:03] poke`: qgil and chasemp are the best call for the case i think [21:28:26] matanya: Thanks, I got a response there :) [21:28:35] great [21:29:27] YuviPanda: http://wikitech-test.wmflabs.org/w/api.php?action=query&list=novainstances&niproject=api-test&format=xmlfm&niregion=eqiad ! [21:29:42] legoktm: yorny [21:29:43] err [21:29:43] yay [21:43:41] andrewbogott: is someone with root on virt* required to deploy to wikitech? or is the cron job that does the deploy in place already? [21:43:52] No cron, I do it by hand [21:44:14] andrewbogott: hmm, if we put the two patches on today's SWAT (in about 1:15), would you be around to do it by hand? [21:44:20] sure [21:44:25] legoktm: ^ [21:44:45] ok, sweet. [21:45:08] I'll prep the cherry-picks [21:50:51] legoktm: thanks! :) [22:50:58] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c9 (10Matthew Flaschen) If it helps, when I do the following, I get a different error: 1. Clear all Beta Labs cookies. 2. Go to http://en.wikipedia.be... [22:52:14] 3Wikimedia Labs / 3deployment-prep (beta): "There was an unexpected error logging in" when creating accounts on Beta - 10https://bugzilla.wikimedia.org/71862#c10 (10Greg Grossmeier) (In reply to Matthew Flaschen from comment #9) > If it helps, when I do the following, I get a different error: > > 1. Clear a... [23:10:14]