[00:05:21] Coren: confirmed still broke on my phone (as a "before" test). let me know when to retest [00:06:10] jeremyb: 30 minutes? [00:06:24] (03CR) 10Jforrester: "Sure, except that it's posting as jenkins-bot – I'm pretty sure we want to hear jenkins's V-1s." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/110830 (owner: 10Jforrester) [00:06:32] scfc_de: idk... it takes more than a merge... [00:06:36] you have to deploy [00:06:45] and then optionally do a puppet run [00:10:32] scfc_de: nvm, i see ori deployed it [00:11:12] jeremyb: Sorry, you're right; the puppet-merge is usually beyond my horizon :-). [00:16:54] The interesting question is which host actually runs dynamicproxy. fastcci1.wmflabs.org resolves to 208.80.153.214 => a) not a Labs instance, b) doesn't appear in Puppet, c) resolves back to fastcci1.wmflabs.org. Hmmm. YuviPanda? [00:17:10] heh [00:27:02] jeremyb: Hmmm. Okay, now I think that the change was 100 % correct, but: install_certificate builds the certificate chain for star.wmflabs.org which apparently is signed by RapidSSL CA, with wmf-labs.pem which is a private CA (that didn't even sign the certificate). *Argl* [00:33:42] As fixing this probably involves having access to the dynamic proxy host and the private key, I'm at a dead-end. I'll update the bug. [00:44:56] scfc_de: check your mail :) [00:47:02] dschwen: ^ [00:47:17] jeremyb: I think such a wide-ranging change would need thorough review. But I thought about a different approach: If we set $ca to "RapidSSL_CA.pem" in install_certificate in manifests/role/labsproxy.pp, that should work without bringing the house down. [00:47:30] ewwww, no [00:47:51] the gerrit commit? [00:47:54] yes [00:48:27] scfc_de: it's definitely wrong as is. where else would even be using the cert? [00:48:44] scfc_de: just tools.wmflabs.org ? [00:49:15] (probably *used* *to* be right. when it used to be self signed) [00:49:21] jeremyb: That's my assumption (and that wouldn't be affected by your change), but I'm not sure. [00:49:32] jeremyb: Probably. [00:50:03] jeremyb: Okay, you have convinced me :-). [00:50:24] well this ssl stuff is giving me a headache ;-) [00:50:36] I'll leave it to the experts [00:51:01] dschwen: i have some ibuprofen here [00:51:32] dschwen: is it normal for https://fastcci1.wmflabs.org/ to be a 500? [00:51:41] yes [00:51:45] try /status [00:52:12] it is a c-server using libonion [00:52:23] i see you don't like my commit msg :P [00:52:46] serves the bare minimum and returns 500 for every invalid or incomplete query parameter set [00:53:18] scfc_de: look at the other commit too! [00:54:06] scfc_de: it's not just "CA". it's an intermediate. and why kill the ref to the parent? [00:56:00] manybubbles: hey javaification man! [00:56:38] jeremyb: apparently I'm a guest. I'm actually about to sign out any way for the evening [00:56:51] Guest39084: good night! [00:57:01] sorry to disappoint [00:57:05] Guest39084: where's the recipe? [00:57:10] Because there's no relationship between the commits? One fixes that chained certificates are served for dynamicproxy, one fixes that the correct ones are created for all of Labs. [00:57:32] scfc_de: there certainly is a relationship... [00:57:48] they were both intended to fix the same problem. [00:59:36] That's referenced by "Bug:", if necessary. IMHO commit messages should describe the change and not be a log (NB: I'm well aware that many @ WMF have a different opinion on that :-)). [01:00:04] so, the tools db replicas are at 192.168.99.* but don't see how traffic is routed there [01:00:45] this is not a log of changes to the gerrit change [01:00:59] it's a link to the previous already merged change [01:01:15] Re planet, it looks alright to me, but I think this is change "all or nothing": If it's the right thing, then do it for all certs. [01:01:29] scfc_de: well i did -1... [01:01:37] There are many already previously merged changes :-). [01:01:52] if someone had bothered to check before merge (which I said I hadn't as a reason for not +1ing) then it would have all been done in a single commit [01:02:32] dschwen: Per iptables. On Tools, look in IIRC /data/project/.system/iptables* and /hosts [01:02:46] oh, ok, that's the pointer I needed [01:03:22] thx [01:04:51] perfect. works. [01:45:38] * jeremyb waits for git... [01:50:14] Slap it and tell it that it looks pretty jeremyb. [01:50:37] T13|sleeps: nah, it finished [01:50:53] Ahh. Good. [09:42:41] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899092 edit summary: /* Active Tools on the Toolserver */ filniks account is expired [09:54:37] Oops, clicked wrong and offered a serious challenge to Visual Editor by making it edit a BIG table. [09:55:37] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899099 edit summary: /* Active Tools on the Toolserver */ [09:58:08] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899101 edit summary: /* Active Tools on the Toolserver */ sahim's account is expired [10:10:25] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899104 edit summary: /* Active Tools on the Toolserver */ Chris' account is expired [10:27:22] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899111 edit summary: /* Active Tools on the Toolserver */ soxred93's account has expired [10:39:38] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899117 edit summary: /* Active Tools on the Toolserver */ W's account has expired [10:45:34] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899121 edit summary: /* Active Tools on the Toolserver */ emijrp's tools are still active on ts [10:46:40] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Steinsplitter link https://www.mediawiki.org/w/index.php?diff=899122 edit summary: Migrated to Labs by Luxo [10:49:01] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899125 edit summary: /* Active Tools on the Toolserver */ diegofb expired [10:51:51] Change on 12mediawiki a page Wikimedia Labs/Tool Labs was modified, changed by Steinsplitter link https://www.mediawiki.org/w/index.php?diff=899130 edit summary: [10:56:57] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899132 edit summary: /* Active Tools on the Toolserver */ mjbmr's tools are active on ts [10:59:28] hashar: hi, is the wikilove tabe confidential? [11:01:04] looks like no :D [11:03:35] Steinsplitter: no idea [11:04:10] Steinsplitter: if it is in the labs replicated databases it is most probably not confidential. If you fnd out there are some private data in there, then we should redact them. [11:18:44] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899136 edit summary: /* Active Tools on the Toolserver */ reza, tools exists in Tool Labs [11:21:16] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899138 edit summary: /* Active Tools on the Toolserver */ piglop active on ts [11:46:29] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/List of Toolserver Tools was modified, changed by Silke WMDE link https://www.mediawiki.org/w/index.php?diff=899140 edit summary: /* Active Tools on the Toolserver */ marked Purodha's tools as migrating though not sure for every single tool [16:33:19] Hi all! [16:33:41] are user databases on the replica servers public or only readable by the project that created them? [16:36:41] dschwen: If I recall correctly, they're public if they end in "_p", or if the user uses GRANT to grant permissions. [16:37:05] the _p was the case on the toolserver [16:37:51] and on labs [16:37:54] found it [16:37:57] thanks [16:45:28] <^d> Coren: Ping [16:51:34] uh oh [16:51:51] I'm trying to load some stored procedures onto the tool labs replica [16:52:10] but I get: ERROR 1548 (HY000) at line 1: Cannot load from mysql.proc. The table is probably corrupted [16:52:19] arwiki.labsdb [16:53:37] 192.168.99.7 [16:53:52] bug or user error [16:54:03] it seems to work fine for the other dbservers [16:54:28] ^d: Pong (in meeting, will be right with you) [16:55:01] <^d> Okie dokie, no rush. [16:59:37] well, bug filed [17:01:38] Yeah, DROP PROCEDURE IF EXISTS ProcName; craps out on sql server arwiki,labsdb [17:01:47] but works fine on commonswiki.labsdb [17:02:02] there is definitely something broken! [17:02:17] https://bugzilla.wikimedia.org/show_bug.cgi?id=60907 [17:28:47] ^d: Back. [17:30:38] <^d> Hi :) So, bug 43652 was filed about being able to perform custom ElasticSearch queries (specifically regex) against search data. This is an awesome idea for labs, we think. I'm wondering if we could look at doing something similar to database replication to real hardware that labs users could then query from. [17:41:51] <^d> Coren: Not easy, I know :p [17:42:15] What backend does ElasticSearch use? [17:42:25] <^d> ElasticSearch is Lucene-based. [17:43:35] ^d: It doesn't seem insane, a priori, but now is about the worst possible time to start something like this (being in the middle of migration) [17:43:49] <^d> Right, I was thinking that. [17:44:39] <^d> I started a page on wikitech: https://wikitech.wikimedia.org/wiki/Search/Labs_services. Maybe we can just work on the details until the migration's over, then look at doing it. [17:49:28] ^d: Sounds like a plan. [17:50:25] <^d> Cool, we'll do that then. Thanks! [19:00:28] Krinkle|detached: Reedy: I need some help on TsIntuition [20:21:18] YuviPanda: Ping [20:21:27] hey TParis! [20:21:32] TParis: can I transition it over now? [20:21:35] Hey - got your message. Yeah, we'd love to have it. [20:21:43] ok, let me get to it [20:21:46] moment [20:24:21] TParis: can you add me to the project as an admin? [20:24:47] TParis: you can un-add me after the transition is done [20:25:48] gah [20:25:51] he left :( [20:26:36] YuviPanda: Wait abort! [20:26:44] TParis: I haven't done anything yet [20:26:52] TParis: you need to make me projectadmin before I can do anything :) [20:27:05] I asked but your client went out just about then [20:27:23] Good deal, I just realized there may be a config change and I'm in the middle of a move from Texas to Hawaii and my main box is on a cargo ship on the ocean with the key I need to log into labs [20:27:38] TParis: ah! :D [20:27:40] TParis: right [20:27:47] I won't get it back for about another month. [20:28:10] It might work, I think we made it protocol insensetive, but I can't be positive [20:28:14] TParis: hmm, so it is about a 5 min bit of change to actually switch over, and then we just need to test. I can do it whenever is most convenient for you [20:29:19] Well, it's up to you, we can try it if you'd like if it's easily reversible. I recall the switch to https on toolserver only having a minor hickup with the reCaptcha and that was resolved so there is a good chance it'll work fine. [20:29:28] TParis: it is easily reversible, so we could try it :) [20:29:31] But I'm not 100% and I dont have my certs on me. [20:29:37] okay [20:30:27] I'll go make you a project admin [20:30:40] TParis: ok! [20:34:10] Hmm, it's showing me that you and Ryan are both listed as project admins [20:35:09] Ahh, I see [20:35:13] TParis: hmm [20:35:18] TParis: i don't see it yet? [20:35:24] Okay, it's done [20:35:39] alright, switching [20:37:12] TParis: https://utrs.wmflabs.org/ [20:37:25] TParis: you should try flushing your DNS cache or incognito [20:37:37] TParis: it should resolve to the IP 208.80.153.214 [20:37:45] TParis: it seems to work fine for me :) [20:38:18] nslookup /dnsflush? [20:38:46] TParis: windows? [20:38:57] yeah [20:38:59] TParis: if so, ipconfig /dnsflush [20:40:04] Still hitting the .185 after a dns flush [20:40:35] TParis: hmm, do you have any other machine around? :) [20:40:58] TParis: try in another browser perhaps? multiple levels of dns cache [20:41:16] No other machine, I'm in a hotel :( [20:41:22] Only brought my laptop [20:41:23] hmm :( [20:41:41] It's alright, the old URL still works for now, the new DNS will resolve itself eventually [20:41:57] http still works* [20:42:11] TParis: hmm, so https doesn't actually work - you have hardcoded several URLs to http :( [20:42:14] [blocked] The page at 'https://utrs.wmflabs.org/' was loaded over HTTPS, but ran insecure content from 'http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js': this content should also be loaded over HTTPS. [20:42:16] and such [20:42:30] Yeah, I thought we fixed all of that [20:42:30] TParis: but http://utrs.wmflabs.org/ will still work, though. [20:42:34] TParis: apparently not [20:42:55] TParis: but the http version will still work - we can just let this be, but just not advertise the https version until those get fixed [20:43:03] Okay, sounds good [20:43:16] TParis: alright! do poke me (here or talk page) if something seems amiss [20:43:28] I'm supposed to get my stuff on the 28th of this month so I'll let you know around then when I am set up [20:43:36] TParis: alright! [20:43:43] TParis: thanks for taking the time now! :) [20:44:46] TParis: anything still on TS moving to labs? [20:45:32] matanya: None of the stuff I maintain. All of Xtools were migrated by Cyberpower and I've shut them down on labs and UTRS has moved here and I've also shut down the labs version. [20:45:41] None of my bots operate anymore either. [20:45:57] good news, thanks TParis [20:51:17] hi TParis I hope you are doing well these days [20:52:19] Hi sumanah, it's been almost a year! I'm doing fairly well, in the middle of a stressful move right now but other than that everything is good. [20:52:59] Moving is just so stressful - my sympathies TParis. Glad everything else is good [22:11:31] Cross-post from #-dev: Does anyone other than hashar have access to integration-meetbot.pmtpa.wmflabs to restart meetbot? Instructions at https://bugzilla.wikimedia.org/show_bug.cgi?id=46377#c6 [22:12:29] bd808: I do, but the machine is dead [22:12:31] anyone know a quick way in puppet to see if a variable is defined? [22:12:35] bd808: let me restart it? [22:12:46] I'd like to fallback and use a default if it's not already defined... [22:13:03] bd808: hmm, nevermind. I don't have enough permissions, apparently. [22:13:14] YuviPanda: Thanks for trying [22:13:20] bd808: :) [22:13:25] bd808: Coren has root keys [22:13:53] cajoel: Undefined vars in puppet == undef [22:15:23] So `if $foo == undef { $foo = 'bar' }` or something like that should work [22:15:52] There is probably a much prettier puppet syntax for this common case [22:20:42] Ah, didn't know we had meetbot. Saw recently minutes for a Fedora meeting and the structure looked great (attendance, action items, etc.). [22:21:57] hedonil: Hi [22:23:05] Krinkle: Hi. i submitted a pull request to TsIntuition. what do I have do to next? [22:24:23] hedonil: Wait for siebrand or myself to review and merge it. After I see the merge and Travis build success, I'll deploy it [22:24:42] bd808: not sute undef is working here.. [22:25:00] Krinkle: ok. An then wait for the translations to be done, I guess. [22:25:14] bd808: barfs it it's in a scope that's not here.. [22:25:16] warning: Scope(Class[Pmacct]): Could not look up qualified variable 'passwords::pmacct::mysqlpass'; class passwords::pmacct could not be found at /home/jkrauska/puppet/modules/pmacct/manifests/init.pp:20 [22:25:37] trying to put in a labs work around if the passwords (private repo) is missing [22:26:12] if $passwords::pmacct::mysqlpass == undef {$mysqlpass = 'testing'} [22:26:46] cajoel: why not use the labs/private repo as that is what it is intended for? [22:26:48] other ideas on how to approach this? (prototying in labs for things that will later depend on private repo?) [22:26:49] cajoel: Ah. One way to fix that particular problem is via the https://gerrit.wikimedia.org/r/#/admin/projects/labs/private repo [22:27:50] cajoel: Although obviously the labs version isn't really private so it may not match your usecase. [22:27:53] I was hoping to make a module that's a little less WMF specific (wouldn't require another repo) [22:28:15] I'll keep digging in to undef-like options.. [22:28:43] seems it's been discussed before but due to order of operations not being consitent, it's been poo-poo'd in the past? [22:28:56] cajoel: I would suggest moving setting the password to a role and make it a required input to the module if you want clean reuse [22:29:30] that sounds useful -- got an example of that type of syntax..? [22:29:44] * bd808 goes to dig something up [22:31:30] another idea that might be nice would be something like if $deployenvironment=='wmflabs' {do this} else {do another thing}.... [22:31:44] do we have a global fact that identifies the deploy environment? [22:31:58] cajoel: That can be done via $::realm [22:32:37] I got lost somethere when puppet was in 0.x land and everything was facter based.. [22:32:43] if $::realm == 'production' { … } else { … } [22:32:47] got a command line way to check $::realm ? [22:33:09] $ facter | grep realm --- no dice. [22:35:22] I'm set... [22:35:40] global defined in realm.pp [22:36:51] I /think/ the puppet master sets it somehow when is sends the manifests down. [22:37:47] wrapping if $::realm worked like a charm [22:37:56] would be happy to also look at your role idea. [22:39:13] We do that quite a bit. For example the logstash module vs the logstash role [22:39:46] Module defines a class or classes that are configurable via parameters and the role fills in specific values [22:40:11] where are the roles defineed [22:40:46] puppet/manifests/role ...