[01:04:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Lolsa112 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=423516 edit summary: [06:40:21] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:03:19] 6Labs, 10Tool-Labs, 10labs-sprint-117, 10DBA: tools.citationhunt can't access databases - https://phabricator.wikimedia.org/T109972#1565044 (10Ladsgroup) I think we should close this bug. [07:03:33] 6Labs, 10Tool-Labs, 10labs-sprint-117, 10DBA: tools.citationhunt can't access databases - https://phabricator.wikimedia.org/T109972#2189305 (10Ladsgroup) 5Open>3Resolved [07:03:57] 6Labs, 10Tool-Labs, 10labs-sprint-117, 10DBA: tools.citationhunt can't access databases - https://phabricator.wikimedia.org/T109972#1565044 (10Ladsgroup) Please re-open if it's still happening. [07:20:23] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:13:29] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:13:45] PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:16:09] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:16:09] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:16:55] PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:17:41] PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:19:19] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:20:25] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:22:05] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:22:17] PROBLEM - Puppet run on tools-exec-1207 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:23:42] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:23:51] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:23:52] PROBLEM - Puppet run on tools-exec-1209 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [08:24:01] PROBLEM - Puppet run on tools-exec-1219 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:24:20] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:25:14] PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:26:24] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:27:32] PROBLEM - Puppet run on tools-exec-1204 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:27:36] PROBLEM - Puppet run on tools-exec-1205 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:28:32] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:32:08] PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:33:32] PROBLEM - Puppet run on tools-exec-1213 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:33:55] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:34:53] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:35:59] PROBLEM - Puppet run on tools-exec-1206 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:38:41] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [08:39:43] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:40:55] PROBLEM - Puppet run on tools-exec-1201 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:26:38] PROBLEM - Puppet staleness on tools-bastion-10 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [12:19:38] 10Tool-Labs-tools-wikiloves: Desenvolver gráficos e tabelas para exibir os dados - https://phabricator.wikimedia.org/T131192#2158781 (10Crang115) Muito bom, Danilo. Olhando rapidamente não me parece ser necessária nenhuma correção. Acho que como você já havia mencionado antes podemos partir para criar novas vis... [13:07:01] RECOVERY - Puppet run on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [13:49:41] (03PS1) 10Ladsgroup: Add #wikimedia-ai channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282364 [13:50:23] legoktm: hey, ^ [13:51:13] (03PS2) 10Ladsgroup: Add #wikimedia-ai channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282364 [13:51:33] (03PS3) 10Ladsgroup: Add #wikimedia-ai channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282364 [16:02:38] 6Labs, 10Horizon: Horizon behavior for non-projectadmins is weird - https://phabricator.wikimedia.org/T132187#2190892 (10Andrew) [16:02:58] 6Labs, 10Horizon: Horizon behavior for non-projectadmins is weird - https://phabricator.wikimedia.org/T132187#2190893 (10chasemp) p:5Triage>3Normal [16:03:26] 6Labs, 10Horizon: Horizon behavior for non-projectadmins is weird - https://phabricator.wikimedia.org/T132187#2190894 (10Andrew) It doesn't help that Horizon has a different policy file for nova than nova has for nova. I think that's still necessary for now though. [16:57:32] Who is a DB admin? [16:58:47] jynus [16:58:51] (not here atm though) [16:59:10] I need to run a heavy query. It's a one time thing [17:09:30] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: Throttling Cyberbot tool user as it is consuming most of the CPU - https://phabricator.wikimedia.org/T131937#2191123 (10Cyberpower678) Just a heads up I'm running ALTER TABLE `s51059__cyberbot`.`externallinks_enwiki` ADD COLUMN `last_deadCheck` INT(10) U... [17:10:39] 6Labs: Make some more public flavors for labs VMs - https://phabricator.wikimedia.org/T132194#2191124 (10Andrew) [17:12:09] 6Labs: Make some more public flavors for labs VMs - https://phabricator.wikimedia.org/T132194#2191137 (10Andrew) [17:45:23] PAWS at Error 502 now [17:45:58] back now [17:46:24] and down again... [17:51:11] PAWS? [17:52:18] 10PAWS: Bad Gateway error - https://phabricator.wikimedia.org/T132196#2191208 (10MarcoAurelio) [17:59:32] ah [18:07:46] (03PS1) 10Mattflaschen: Echo was renamed to Notifications [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282403 [18:15:20] (03CR) 10Luke081515: [C: 031] Echo was renamed to Notifications [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282403 (owner: 10Mattflaschen) [18:17:53] 6Labs: Make some more public flavors for labs VMs - https://phabricator.wikimedia.org/T132194#2191309 (10chasemp) p:5Triage>3High [18:23:58] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: Throttling Cyberbot tool user as it is consuming most of the CPU - https://phabricator.wikimedia.org/T131937#2191339 (10Volans) @Cyberpower678 I see that your table has a quite big composite primary key (8 fields including a `varchar(767)`) and various ad... [18:27:23] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: Throttling Cyberbot tool user as it is consuming most of the CPU - https://phabricator.wikimedia.org/T131937#2191342 (10Cyberpower678) >>! In T131937#2191339, @Volans wrote: > @Cyberpower678 I see that your table has a quite big composite primary key (8 f... [19:08:21] 10PAWS: Bad Gateway error - https://phabricator.wikimedia.org/T132196#2191464 (10MarcoAurelio) p:5Triage>3Unbreak! Since it's a service that worked in the past and stopped being avalaible out of the blue. Feel free to lower, though. [19:15:38] 6Labs, 10PAWS, 10Tool-Labs: Bad Gateway error - https://phabricator.wikimedia.org/T132196#2191489 (10MarcoAurelio) [19:17:14] mafk: looking at it now [19:17:40] oh YuviPanda I didn't noticed you were here [19:17:44] thanks [19:18:13] I just got here [19:18:32] and I'm just out for dinner :D [19:18:35] see you [19:19:59] 6Labs, 10PAWS, 10Tool-Labs: Bad Gateway error - https://phabricator.wikimedia.org/T132196#2191208 (10Krenair) Domain is a proxy to http://paws-proxy-01.paws.eqiad.wmflabs:80 which is what is returning the 502 Bad Gateway [19:20:48] I could probably debug that further if I had access to the paws project [19:20:51] but nope [19:22:13] oh but YuviPanda does, ok [19:22:15] Krenair: it's actually just in tools [19:22:25] Krenair: the paws project is just for the domain. [19:22:30] Don't think I hold admin in tools anymore [19:22:38] hub-7o4lp 0/1 CrashLoopBackOff 33 1d [19:22:57] what is that from? [19:23:08] kubernetes, which is where this is running in [19:23:12] Ah. [19:23:14] kay, [19:23:19] maybe I couldn't help then. :) [19:27:32] 6Labs, 10PAWS, 10Tool-Labs: Bad Gateway error - https://phabricator.wikimedia.org/T132196#2191502 (10yuvipanda) a:3yuvipanda [19:29:18] 6Labs, 10PAWS, 10Tool-Labs: Bad Gateway error - https://phabricator.wikimedia.org/T132196#2191208 (10yuvipanda) 5Open>3Resolved Hmm, I just restarted the pod, and it went away. I also got paged for it, so not too bad :D [19:42:56] 6Labs, 10Tool-Labs: Goal: Allow using k8s instead of GridEngine as a backend for webservices (Tracking) - https://phabricator.wikimedia.org/T129309#2191566 (10yuvipanda) [19:50:57] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2191580 (10yuvipanda) a:5yuvipanda>3None [19:54:41] 6Labs, 10Labs-Infrastructure: Labs proxy api (aka 'Invisible Unicorn') is a spof - https://phabricator.wikimedia.org/T131308#2191585 (10yuvipanda) https://github.com/otoolep/rqlite is an interesting option, since our write / read load is fairly tiny. [20:26:42] Good evening everyone [20:27:08] I wondered if someone could help me [20:27:13] I have a few friends coming over tomorrow for a micro hackathon in my kitchen, we checking if lab access is working (SSH) and having issues for for Nicko777 [20:27:21] Good evening. [20:27:22] wikitech username=nicko [20:27:46] Got some issue trying to connect to labs servers [20:28:12] What's the issue? [20:28:39] I had my request completed, my public key is at the right place (under the openstack tab) [20:28:57] ~ $ ssh sonarqube-poc.eqiad.wmflabs Permission denied (publickey). ssh_exchange_identification: Connection closed by remote host [20:29:08] And you can't log in to labs? [20:29:08] * YuviPanda waves at people [20:29:16] after having entered my passphrase [20:29:19] Nicko777: can you paste your .ssh/config? [20:29:24] sure [20:29:24] Nicko777 actually can't even log in to bastion [20:29:48] Being able to do that helps. [20:29:49] how can I format it ? [20:29:50] Host * UseRoaming no Host bastion.wmflabs.org ProxyCommand none ControlMaster auto Host *.wmflabs.org *.wmflabs User Nicko IdentityFile ~/.ssh/id_rsa ProxyCommand ssh -a -W %h:%p bastion.wmflabs.org [20:29:56] YuviPanda: let me know what you check so I can debug it myself next time... [20:30:03] * gehel love learning new tricks... [20:30:32] gehel: ok, so step 1 is to login to bastion yourself (you need to do this as root, since normal ops members aren't allowed to get into normal bastions) and look at auth.log [20:31:18] Nicko777: can you try 'ssh bastion.wmflabs.org' now? [20:31:30] gehel: now I'm tailing auth.log on bastion.wmflabs.org [20:31:33] YuviPanda: damn, for that I would need to root pwd, I dont have access to pwstore yet (never needed it so far). Let me put that on my todo... [20:31:50] gehel: ah, no you don't. it's labs so no root pw. we can just add your key to root keys. [20:32:01] tried got exactly the same issue [20:33:29] gehel: https://github.com/wikimedia/labs-private/blob/master/modules/passwords/templates/root-authorized-keys.erb can you find that file in puppet and add your key there? and I'll merge it [20:33:47] YuviPanda: wilco [20:33:53] Nicko777: ok, try it once more? [20:34:28] still the same [20:34:47] > Apr 8 20:34:14 bastion-01 sshd[7236]: input_userauth_request: invalid user nicko [preauth] [20:34:48] should I paste the verbose output ? [20:34:56] Nicko777: nope ^ so the problem is your username [20:35:13] Nicko777: in wikitech you can find your 'shell username' under preferences. can you see what that is? [20:35:17] my username has a capital letter as first letter [20:35:32] Nicko777: indeed, but that's the wikitech username, not the shell one that's in LDAP [20:35:58] mmmh got it, my bad [20:36:03] I'll try [20:36:05] again [20:36:35] Nicko777: np, this is confusing :) [20:36:44] much better with the right username [20:36:46] sorry again [20:36:52] bd808: ^ another thing we can make easier when we no longer have to use wikitech [20:36:53] and thanks for your help [20:37:02] Nicko777: np! and thanks for contributing :D [20:37:10] gehel: ^ see debugging process :) [20:37:43] Another pain, there are 2 usernames, one for shell, and one for wikitech. [20:37:59] indeed [20:38:04] yup. its not the most awesome thing [20:39:01] I can help a bit with Windows stuff, because that's hellish. [20:39:13] damn, my wikitech username and my shell username are the same, I did not remember they could be different [20:39:33] YuviPanda: Thanks! I owe you one! [20:39:46] (03PS1) 10Gehel: Adding SSH key for gehel / glederrey@wikimedia.org [labs/private] - 10https://gerrit.wikimedia.org/r/282428 [20:39:48] Mustn't forget the capitalisation of them as well. [20:40:14] good night [20:40:22] Night. [20:40:23] gehel: is that the same as your prod key, btw? Try not to use same keys for labs/prod [20:40:35] (I don't know if it is yours, just standard disclaimer etc) [20:40:36] nope labs specific [20:40:52] (03CR) 10Yuvipanda: [C: 032 V: 032] "Welcome!" [labs/private] - 10https://gerrit.wikimedia.org/r/282428 (owner: 10Gehel) [20:41:03] YuviPanda: thanks again! [20:41:07] gehel: cool, you'll have root on all labs instances once next puppet runs [20:41:29] I'll try to make good use of that! [20:41:55] night! [20:42:09] gehel: night! [20:57:53] YuviPanda, andrewbogott: is there any way to script creating Labs proxies? [20:58:06] subbu needs to make a LOT of them [20:58:43] a lot == 82 apparently [21:00:29] well, there is the painful way of using selenium ide plugins to record clicking things, then adjust the scripts to keep changing the dns. but thats a pain and hopefully there is a better way [21:00:49] bd808: so there's an api [21:01:10] bd808: invisible-unicorn.py in operations/puppet. you can hit it from silver or novaproxy-01.eqiad.wmflabs [21:01:33] bd808: and then we'll have to script addition of all of these records into designate, which has to happen on labcontrol1001 [21:02:16] so 1 script for nginx and one for dns basically? [21:02:19] YuviPanda: is that how horizon does it, adds to both locations? [21:02:50] chasemp: yeah. it hits the API for nginx, and designate for DNS [21:02:54] bd808: yeah [21:03:20] gotcha [21:03:23] Is there a way to make a wildcard record for either part? [21:03:56] the use case is 2 Labs vms that each need 41 hostnames for emdiawiki-vagrant vhosts [21:04:10] so *.foo.wmflabs.org would work [21:04:14] if it is going to be more work writing these scripts than filling out those forms .. i have 76 more to do .. I will do it .. unless there is value in writing these scripts for the future. :) [21:04:37] *-mw-base.wmflabs.org and *-mw-expt.wmflabs.org [21:05:02] dns doesn't really do wildcards afaik [21:05:16] changing the first `-` to a `.` would be an easy config change on the mw-vagrant side [21:06:08] andrewbogott is best suited to answer honestly, anything I venture is a guess for the openstack side of things [21:06:11] chasemp: dns supports *.example.com (domain wildcard) [21:06:14] yes .. i can use whatever the format is. i don't need that - form necessarily. [21:06:38] the problem with *.$Projectname.wmflabs.org is... SSL [21:06:44] right [21:06:52] but for this not too worrysome [21:07:00] actually let's step completely back [21:07:04] subbu is just doing visual diff testing [21:07:16] subbu: what are you trying to do? is this a beta type setup? [21:07:22] bd808: yeah you are right of course :) I was thinking more like *-thing.com [21:07:30] subbu: and do you need SSL? [21:07:42] no. [21:08:04] subbu: are they all proxying back to one instance or? [21:08:05] no user sessions and i'll probably make both mw installs read only. [21:08:27] yes .. 2 vms running multi-wiki mediawiki [21:08:35] subbu: ok. here's what I think we should do: I'll give you a floating IP, and we can set *.$projectname.wmflabs.org to point to that floating IP on DNS. and then you can proxy from there as you wish. [21:08:58] this would be much easier in the long run than maintaining a large number of proxies, since you don't need SSL [21:09:48] i am not familiar with writing proxy scripts .. :( .. so, you mean write a nginx script? [21:09:57] s/script/config/ [21:10:03] subbu: yeah. I can probably help with that if you want. [21:10:26] they we can setup nginx on the vm that the points to to reverse proxy the right mw-vagrant host [21:10:29] bd808, do these vms already have nginx? [21:10:35] bd808: yup [21:10:36] s/they/then/ [21:10:55] subbu: probably the easies thing to do is setup a small instance just to be the proxy [21:11:09] * YuviPanda agrees with bd808 [21:11:13] base install + manually install and configure nginx [21:11:32] it will be just a single .conf file, so you can just put it in a random git repo somewhere [21:11:39] or just a wiki page even [21:11:50] https://www.nginx.com/resources/admin-guide/reverse-proxy/ [21:12:37] if this seems acceptable, let me know and I'll get y'all started [21:13:15] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: Throttling Cyberbot tool user as it is consuming most of the CPU - https://phabricator.wikimedia.org/T131937#2191764 (10Cyberpower678) So the table has finished altering. @Volans I would appreciate your input on what I should on the table to make it bett... [21:13:19] * subbu just wants the end result .. ;-) fine with whatever solution seems best. [21:13:32] sounds right to me. I can help get it setup [21:13:58] thanks you both. [21:14:12] the only thing I don't know yet is how to filter the hostnames. lua magic? [21:14:29] bd808: filter how? [21:14:51] subbu: what's the name of the project? also can you file a task stating the end result you want (mostly just what we talked about) [21:14:54] half of the hostnames need to go to instanceA and the other half to instnaceB [21:15:05] bd808, YuviPanda you find this approach useful to figure out process for potential future needs, right? [21:15:08] project is wikitextexp [21:15:26] if not, spending an hour filling out web forms is not the end of the world. [21:16:20] yes, wikitextexp is the project. [21:16:38] subbu: indeed, but htis is also similar to how beta is setup, and if someone has this question in the future we can just point them back to ticket :D [21:16:56] okay then. i'll create the ticket. [21:17:10] subbu: filling out the web form is also negative in other aspects - imagine one of those instances dies, now you have to manually fill out a bunch of clicks again :) and again... [21:17:11] against which project? labs? [21:17:21] YuviPanda, true. [21:17:30] subbu: yup [21:17:32] I'm spinning up wikitextexp-proxy as a jessie instance [21:17:32] k [21:18:20] !log wikitextexp increased floating ip quota to 1 [21:18:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikitextexp/SAL, Master [21:18:38] bd808: if the list is known you can just put it in the config itself I think. [21:20:30] * andrewbogott is sort of back [21:20:41] I haven't read the backscroll yet, but, subbu, why not a wildcard? [21:21:41] andrewbogott: if setting up a wildcard in horizion is easy then yeah that would work great [21:21:42] then we'd only need 2 [21:21:42] ah, I see that YuviPanda already got there :) [21:21:46] andrewbogott, I don't know how the pieces all fit together and what is allowed and what is not where .. so I cannot answer that question. :-) [21:22:58] I think my internet is being flakey :/ [21:23:33] andrewbogott: btw, I'm trying out the CNAME stuff now. [21:23:41] 6Labs: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2191770 (10ssastry) [21:23:47] there ya go ^ [21:27:30] subbu: you're running one apache on each system, with a bunch of vhosts? [21:27:40] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2191790 (10yuvipanda) a:3yuvipanda I just tested it, and it works. Going to convert a bunch of 'em now. [21:28:09] andrewbogott, I just spun up a VM. nothing more. didn't look into what those labs vms are running. [21:28:09] andrewbogott: I think bd808 is setting up a simple proxy host that we can point a wildcard DNS at with a floating iP, and it'll partition based on hostname [21:29:03] andrewbogott: yes, apache and vhosts. He's got mediawiki-vagrant installed with a lot of wikis enabled [21:29:25] YuviPanda: ok. I feel like even that is more complicated than we need…. why not just two IPs, and one wildcard DNS entry for each IP? [21:29:31] T120345 [21:29:31] T120345: Set up mass visual diff testing with a custom install of mediawiki - https://phabricator.wikimedia.org/T120345 [21:29:37] Then you can have exactly as many sites on each host as you care to write vhsots [21:30:15] 2 ips would be ideal for this one [21:30:26] andrewbogott: that's valid too, if they just want roundrobin distribution and not specifics [21:30:32] then we wouldn't need the proxy [21:30:48] *.apples.wmflabs.org routes to mw-base.wikitextexp and *.oranges.wmflabs.org routes to mw-expt.wikitextexp and then after that it's all bd808's problem :) [21:30:53] YuviPanda: any reason not to do that? [21:31:04] andrewbogott: I think they want it to be not apples and oranges but just apples. [21:31:18] no, apples and oranges would be fine [21:31:52] *.mw-base and *.mw-expt would be perfect [21:31:55] oh? cool then yeah I think that's the easiest thing to do. it would mean that adding new extra machines will require ops to increase floating IP quota. [21:31:57] cool, then you just need me to set the floating IP quotas and you should be able to do everything else in Horizon. [21:32:18] project is wikitextexp right? [21:32:22] yes [21:33:19] ok, raised floating ip quota to two [21:33:38] let's see, I guess I need to make these domains too... [21:33:48] YuviPanda: want to do it, just so you've done it? [21:34:07] andrewbogott: yeah, let me get on labcontrol now [21:34:17] actually you have to do it on californium [21:34:22] oh? [21:34:25] ~root/makedomain [21:34:42] interesting. [21:34:52] yeah, because the designate client for kilo is lousy [21:35:03] and californium is the only prod box with liberty packages [21:35:17] (which means in the near future makedomain will be on labcontrol1001) [21:35:25] andrewbogott: I am there now. what do I run ? :D [21:35:41] can we just make subdomains using horizon? [21:35:57] hm, apparently that script is broken — sorry yuvi, hang on [21:36:24] bd808: no, because creating a subdomain of a parent domain requires permission from the project that owns the parent [21:36:49] but we own wikitextexp.eqiad.wmflabs already right? [21:37:38] andrewbogott: yeah, can't it be something.wikitextesp.wmflabs.org? [21:37:41] so could we in theory make *.mw-expt.wikitextexp.eqiad.wmflabs using horizon only? [21:37:47] YuviPanda: ok, try sudo su - and then ~/makedomain [21:38:09] bd808: I have the gui for domain creation disabled on horizon because most of the time it will fail [21:38:16] *nod* [21:38:31] and it seems better to just say "ask an op to create a domain" than "Go ahead and try! Probably it won't work, due to invisible interactions with projects you aren't in" [21:38:34] ok [21:38:48] --project wikitextexp [21:38:50] the "associate floating ip" action in horizon seems to be spinning forever [21:39:15] and domain is base.wikitextexp.wmflabs.org and expt.wikitextexp.wmflabs.org? [21:39:29] yeah, but probably needs a terminal . [21:39:54] YuviPanda: that script is only barely tested, may fail dramatically [21:40:36] andrewbogott: AttributeError: 'NoneType' object has no attribute 'find' [21:40:44] on ./makedomain --project wikitextexp --domain base.wikitextexp.wmflabs.org. [21:41:02] YuviPanda: ok, I'll try [21:41:18] andrewbogott: ah, needed to source novaclient first [21:41:23] andrewbogott: now at raise exceptions.from_response(resp, method, url) [21:41:25] keystoneclient.exceptions.BadRequest: Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-40cfd4ac-93cd-472a-97f8-95d930b7a230) [21:41:52] hm... [21:42:06] ok, so, I guess the ownership restriction applies to subdomains but not subsubdomains... [21:42:16] sorry, this is all pretty new :) [21:42:26] YuviPanda: in that case you can probably just do this with 'openstack zone create' [21:42:33] but also I can do it since clearly the process is messy [21:44:08] andrewbogott: +1 :D [21:44:14] ok [21:48:26] andrewbogott: how do I associate a floating ip with an instance? the command in https://horizon.wikimedia.org/project/instances/ only times out [21:49:48] bd808: what subdomains do you want me to create? [21:50:12] base.wikitextexp.wmflabs.org and expt.wikitextexp.wmflabs.org wold work [21:50:36] and then we want * A records for each [21:51:39] bd808: I'm not sure about binding the IP — that's worked for me in the past [21:52:02] looks like I sent you guys off on an interesting ops problem. :) [21:52:22] It stopped timing out, but now just says "Error: Unable to associate floating IP." [21:52:55] oh. the ips now show as used in the overview [21:53:19] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2191822 (10yuvipanda) I have cleaned up some bastion-* that were pointing to things redundantly. [21:53:36] bd808: there's another gui for managing floating IPs in Access & Security [21:54:50] andrewbogott: cool. apparently the timeouts actually assocaited ips [21:55:01] and both ended up on the same host [21:55:03] fixing now [21:55:22] oops, I'm stepping on your toes then [21:55:23] sorry [21:55:51] no worries. [21:55:59] * YuviPanda waits for there to be less contention for andrewbogott before asking his arrat y of questions [21:56:01] ips assigned now [21:56:18] and I created the wildcard records — do they work as you'd expect? [21:57:15] both are resolving to 208.80.155.182 for me [21:57:45] ok, how about now? [21:57:55] (I'm just twiddling things in horizon) [21:59:01] looks right now [21:59:09] * bd808 was fiddling too [21:59:39] subbu: I think we just need to fix up your VMs now [22:00:34] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2191845 (10yuvipanda) There's only three bastion domains left: 1. bastion.wmflabs.org 2. bastion2.wmflabs.org 3. bastion-restricted.wmflabs.org These should CNAME to primary.bastion.... [22:00:36] sure .. what is involved? dumps are importing right now. so, if it involves restarting, not now. :) [22:00:38] andrewbogott: let me know when available :D [22:00:44] YuviPanda: now is good [22:01:04] andrewbogott: ok, so there's no domain for bastion.wmflabs.org I Can find, so I can't create $.bastion.wmflabs.org [22:01:26] andrewbogott: do domains get created when projects get created? [22:01:31] *.bastion.wmflabs.org? [22:01:34] andrewbogott: yeah [22:01:39] subbu: no restart needed. It will be a hiera setting to give the new domain name to expect [22:01:42] bd808, i should asked this long back .. and I assume the answer is yes ... vagrant dbs are preserved between vm reboots, right? :) [22:01:53] andrewbogott: not a wildcard domain, but just primary.bastion.wmflabs.org etc [22:01:58] bd808, sounds good. you can tweak any time then. [22:02:00] root@californium:~# designate --os-tenant-name bastion domain-list [22:02:04] is empty [22:02:06] subbu: yes. db is kept until you do `vagrant destroy` [22:02:11] ok. [22:02:32] * YuviPanda salts a 'vagrant destroy' [22:02:42] * subbu wont be touching the vagrant destroy command anytime in the immediate future. [22:03:08] andrewbogott: that's true for other projects too (like telnet) [22:03:34] YuviPanda: this is a bit complicated, but not very much is automated. [22:04:00] I created project-specific subdomains for a bunch of projects when we switched over. [22:04:31] But, when we want to have an A-record for foo.wmflabs.org AND a subdomain foo.wmflabs.org things get confusing [22:04:51] https://phabricator.wikimedia.org/T131367 [22:04:59] That's not exactly the issue but it's similar [22:05:18] right. [22:05:37] so bastion.wmflabs.org has a record in the wmflabs.org project [22:05:48] and, ergo, there's no bastion.wmflabs.org subdomain. [22:06:08] I can create one, and move the bastion.wmflabs.org record to that subdomain. That will… probably not cause an outage :) [22:06:49] andrewbogott: so I want to move bastion.wmflabs.org to a CNAME and also create primary.bastion.wmflabs.org etc. so that's two changes that need to happen, I guess :) [22:07:00] andrewbogott: > move the bastion.wmflabs.org record to that subdomain. [22:07:07] what do you mean by that? [22:07:19] andrewbogott: I'm wondering if I should just let you handle https://phabricator.wikimedia.org/T131796 :D [22:07:26] YuviPanda: that's part of that ticket above [22:07:28] instead of playing a game of telephone [22:07:38] right now bastion.wmflabs.org is a record on wmflabs.org [22:07:48] but once there's a bastion.wmflabs.org domain, that won't work [22:08:02] can a bastion.wmflabs.org domain have a bastion.wmflabs.org record? [22:08:08] yes [22:08:45] want me to do that? [22:08:49] thinking [22:09:19] andrewbogott: there's also bastion2.wmflabs.org and bastion-restricted.wmflabs.org. can bastion.wmflabs.org. domain have records for these? I'm guessing 'no' [22:09:33] indeed not [22:09:53] we could make a subdomain 'bastionhosts.wmflabs.org' and then there would be no conflict [22:09:54] but, can we create bastion2 and bastion-restricted domains, and transfer ownership of those to the bastion project? [22:10:04] ah, I see. [22:10:43] andrewbogott: I'm not sure that's the best long term solution, since I suppose we'll eventually want people to just use primary.bastion.wmflabs.org etc rather than the first level domain [22:11:19] yeah, there's no real reason we can't do everything under bastion.wmflabs.org. It just requires a lot of commandline magic [22:12:38] andrewbogott: can the bastion project 'own' the bastion.wmflabs.org domain? [22:12:44] yes [22:12:58] but things like bastionfoo.wmflabs.org will still stay in the wmflabs.org project [22:13:35] andrewbogott: ok, so can we 1. create bastion.wmflabs.org domain, 2. transfer bastion.wmflabs.org A record to it, 3. Make the bastion project owner of bastion.wmflabs.org [22:14:12] as first steps. Then can we do the same for bastion-restricted? I'm going to kill 'bastion2'. [22:14:12] steps 1 and 3 are what 'makedomain' is about. [22:14:53] hmm, maybe I can actually kill bastion-restricted [22:15:02] really? [22:15:06] only ops need to use it, and we can just as well use restricted.bastion.wmflabs.org [22:15:10] yeah [22:15:13] true [22:15:29] it's not even labs admins - just ops, and we're fairly easily reachable [22:15:37] andrewbogott: so I think that's a valid solution. [22:15:51] so we just need to do (1) (2) (3) for bastion :) [22:18:12] andrewbogott, when might https://phabricator.wikimedia.org/T129181 be done, you think? [22:19:59] subbu: I'm not sure, I don't immediately know how to do it. It should be possible though [22:20:35] 6Labs: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2191882 (10bd808) @Andrew and @yuvipanda allocated 2 floating IPs to the project and setup a wildcard DNS record for each: `*.base.wikitextexp.wmflabs.org` and `*.expt.w... [22:22:19] andrewbogott, ok .. that is a blocker for being able to access the test results without logging into the vm ... basically it would look something like http://parsoid-tests.wikimedia.org/ which makes it convenient to share results and investigate failures etc .. it is not critical in any way and our work is not blocked on that .. but mostly telling you why it is useful. :) [22:23:53] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2097345 (10yuvipanda) This requires adding a DNS entry for promethium.wmflabs.org (since we've SSL only for *.wmflabs.org) pointing to the proxy, and then manually hitting the proxy API with the target IP.... [22:24:23] bd808, reg https://phabricator.wikimedia.org/T132216#2191882 are they mw-base.wmflabs.org and mw-expt.wmflabs.org .. ? [22:24:58] subbu: nope. we left the mw- off [22:25:10] http://en.expt.wikitextexp.wmflabs.org:8080/wiki/Main_Page [22:25:38] k [22:25:52] that is actually cleaner anyway :) [22:27:47] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2191959 (10yuvipanda) a:5yuvipanda>3Andrew After talking with @andrew, this requires the following steps: 1. Remove current bastion.wmflabs.org record from wmflabsdotorg project 1... [22:32:30] 6Labs: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2191962 (10bd808) Mediawiki-Vagrant in Labs is setup to expect a reverse proxy in front of it that maps port 80/443 to 8080 on the host. MediaWiki is configured to gener... [22:36:21] YuviPanda: ok, bastion.wmflabs.org should now look the way you'd expect it [22:37:08] …assuming I got the IP right [22:38:04] andrewbogott: looks ok! [22:39:30] andrewbogott: I'm setting up the other stuff now [22:43:33] (03CR) 10Catrope: [C: 032] Echo was renamed to Notifications [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282403 (owner: 10Mattflaschen) [22:44:05] andrewbogott: ok, I've setup restricted.bastion.wmflabs.org and sent out email. it already works now, so do switch if you can :) [22:45:52] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2191974 (10yuvipanda) 5Open>3Resolved a:5Andrew>3yuvipanda Ok, andrew has setup the changes. I've sent out email about bastion-restricted being deprecated to the ops list. I've... [22:47:12] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2191977 (10yuvipanda) 5Resolved>3Open [22:47:13] YuviPanda: yep, works fine [22:47:36] andrewbogott: hmm, I don't know if bastion.wmflabs.org can be a CNAME. I can't seem to change the type, so do I have to delete and recreate? [22:48:05] oh... [22:48:08] hm [22:48:26] I can change it — what should it point to? [22:49:06] (03Merged) 10jenkins-bot: Echo was renamed to Notifications [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282403 (owner: 10Mattflaschen) [22:49:33] YuviPanda: ^ [22:50:05] andrewbogott: primary.bastion.wmflabs.org as a CNAME [22:51:33] !log tools.wikibugs Updated channels.yaml to: 2fc4b94daebe62f8e5e8712d753fabb5878e418a Echo was renamed to Notifications [22:51:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [22:51:37] 6Labs, 10Tool-Labs: Add SSHFP dns records to bastions - https://phabricator.wikimedia.org/T132225#2191980 (10yuvipanda) [22:53:28] welllllllll [22:53:54] this is a bug, designate thinks that pointing bastion.wmflabs.org to primary.bastion.wmflabs.org is a circular reference [22:54:14] so it's back to being an A record for now [22:55:06] andrewbogott: ok [22:55:23] andrewbogott: I've switched all references to bastion.wmflabs.org in wikitech to primary.bastion.wmflabs.org [22:55:36] andrewbogott: I'll send out a deprecation email [22:56:01] great [22:56:12] we don't need to really shut it off for, like, a year. [22:57:00] andrewbogott: cool. I'm going to actually not announce it right now - we can do it as part of the tools bastion changes [22:57:51] andrewbogott: I think I can call the bastion parts done for now. [23:12:09] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2192050 (10yuvipanda) It turns out you can't CNAME bastion.wmflabs.org while having A records for subdomains, so we'll leave it as is for now. We'll email about the change from bastion... [23:16:48] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2192056 (10yuvipanda) a:3yuvipanda [23:16:56] subbu: ^ am going to do that now [23:20:42] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2192077 (10yuvipanda) Created DNS entry for mw-expt-tests.wmflabs.org [23:21:42] 6Labs, 15User-bd808: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2192078 (10bd808) 5Open>3Resolved a:3bd808 Setup reverse proxy using nginx: ``` $ sudo apt-get install nginx-light $ sudo vim /etc/nginx/sites-enab... [23:24:49] 6Labs, 15User-bd808: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2192083 (10yuvipanda) @ssastry can I remove the proxies you added by hand? :D [23:31:24] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2192085 (10yuvipanda) I was able to add the proxy with: ``` curl -X PUT -d '{"domain": "mw-expt-tests.wmflabs.org", "backends": ["http://promethium.wikitextexp.eqiad.wmflabs:80"]}' localhost:5668/dynamicpr... [23:34:27] 6Labs: Add DNS entry for promethium.wikitextexp.eqiad.wmflabs - https://phabricator.wikimedia.org/T129181#2192089 (10yuvipanda) 5Open>3Resolved Except somehow the domain name doesn't want to resolve for nginx, despite it resolving fine for dig. an nginx restart didn't help either. So I've just used the IP in... [23:35:35] I think that's it for me today everyone. have fun [23:40:24] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2192092 (10yuvipanda) List of domains leftover on wmflabsdotorg: ``` huggle-rc.wmflabs.org. tools-checker.wmflabs.org. tools-bastion-mtemp.wmflabs.org. tools-dev.wmflabs.org. tools-tr... [23:41:43] 6Labs, 15User-bd808: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2192093 (10ssastry) >>! In T132216#2192083, @yuvipanda wrote: > @ssastry can I remove the proxies you added by hand? :D Yes. :) [23:42:01] 6Labs, 10Tool-Labs: Convert most top level tool and bastion dns redcords to CNAMEs - https://phabricator.wikimedia.org/T131796#2192106 (10yuvipanda) That was after I deleted a bunch of domains - mwds-proxy (mwds is long over), matterirc (RIP) and tools-docker-registry.wmflabs.org (is docker-registry.tools.wmfl... [23:45:24] 6Labs, 15User-bd808: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2192110 (10yuvipanda) Done. [23:46:08] 6Labs: Cleanup proxies that point to nonexistent instances - https://phabricator.wikimedia.org/T132231#2192111 (10yuvipanda) [23:46:50] !log bastion terminated mwds-proxy instance + floating IP [23:46:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion/SAL, Master [23:47:25] so much for 'I am done for the day' [23:55:06] 6Labs, 10Tool-Labs: Make catchpoint hit checker.tools.wmflabs.org not tools-checker.wmflabs.org - https://phabricator.wikimedia.org/T132233#2192147 (10yuvipanda)