[00:00:10] chasemp: the merge conflict is probably still causing problems on wikimetrics, but I think what you did with git killed my commit there anyway [00:00:19] I tried grepping in git log and couldn't find it [00:00:32] sorry man I didn't remove anything? [00:00:35] check stash? [00:00:35] so I'd have to fix that manually anyway I think, for which I saved the passwords file [00:00:43] my first move would be a git stash i [00:00:46] nono, it's fine, I can fix it manually [00:01:08] oh, no, nothing would be in the stash, we just keep a local commit as the HEAD with whatever custom changes [00:01:17] I should've told you, that's my bad [00:01:37] but my point is, chasemp: you can do whatever you like on that server and I'll restore it back to normal [00:01:40] ok [00:01:45] so reset / pull / whatever [00:02:47] I'm in your server now dude [00:03:11] I know, I'm just following along with git status [00:03:24] all up in your server [00:03:31] lol [00:03:47] I'm stalking you, man, I'm not sure which one of us will get uncomfortable first [00:05:06] ok try to ssh and sudo now? [00:07:19] YuviPanda btw: https://phabricator.wikimedia.org/T120891 [00:07:42] works chasemp [00:07:49] thank you very much [00:07:54] I do not understand limn1 tbh atm [00:08:07] it's behind but pull says updated [00:08:10] I don't get it [00:09:44] chasemp: you tried a git pull --rebase? [00:10:14] milimetric: I gotta run in like 2 but try root@limn1.analytics.eqiad.wmflabs I dropped you in for the moment to do your thing [00:10:35] is this a thing that needs to figured out tonight? [00:11:12] chasemp: i'll poke at it, no worries [00:11:21] I'll grab someone else to help, have a good night [00:11:51] I'll check back in a few when I can but also you should be able to fix and run puppet as root for the moment [00:11:55] thanks [00:16:25] chasemp: it used to be horribly written, I rewrote it :D [00:17:51] chasemp: we should make it easy for people to drop their own keys into root [00:24:05] YuviPanda: limn1 has: Error 400 on SERVER: Could not find resource 'Exec[compile puppet.conf]' for relationship on 'Class[Puppetmaster::Ssl]' on node limn1.analytics.eqiad.wmflabs [00:24:30] the operations/puppet repo is up to date [00:24:53] 6Labs: 'virt1' entry at markmonitor? - https://phabricator.wikimedia.org/T102689#1864257 (10RobH) I asked Doni to provide the listing, since it seems impossible they are unable to change something and cannot tell why they cannot change it. Since this means someone has to grep every single MM domain for the info... [00:25:02] milimetric: are you applying any roles via the default site.pp? [00:28:04] YuviPanda: the only thing in the custom commit is "import 'passwords.pp'" [00:28:07] (in site.pp) [00:28:13] hmmm [00:28:47] git diff HEAD^ is basically that, the manifests/misc/limn1.pp file and a bunch of custom templates [00:29:25] so other self hosted puppetmasters work fine and this doesn't... [00:29:31] I'm not particularly sure what's going wrong here [00:29:41] well they don't work fine until puppet is run again [00:29:53] yah but that particular error has been going on for weeks [00:29:54] but when chase logged in and pulled with root, puppet ran on wikimetrics1 [00:30:02] oh really? it's still the same one? [00:30:10] never mind then, just ignore it, hasn't caused problems yet [00:30:17] sorry, thought it was something new [00:30:25] * milimetric out [00:30:26] :) [00:30:29] enjoy your night [00:30:42] milimetric: I just came to the office :D [00:30:45] chasemp: don't worry, everything's fine [00:30:46] lol [00:30:54] :D [00:30:54] enjoy your... breakfast then? [00:31:02] I did indeed have breakfast just now [01:14:40] YuviPanda, hey [01:14:46] I can't get into deployment-salt [01:14:49] or deployment-restbase01 [01:15:31] andrewbogott, ^ [01:15:54] am looking [01:15:58] at deployment-salt [01:16:12] YuviPanda: did you get chase’s earlier email about the ldap cert mismatch? [01:17:04] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class puppet::self::geoip for deployment-salt.deployment-prep.eqiad.wmflabs on node deployment-salt.deployment-prep.eqiad.wmflabs [01:17:06] Warning: Not using cache on failed catalog [01:17:08] Error: Could not retrieve catalog; skipping run [01:17:44] that’s not what I expected [01:18:08] oh, I guess broken puppet = not updating ldap = no logins [01:18:29] YuviPanda: so probably if you change /etc/ldap.conf you can get Krenair access if you don’t want to fix puppet yourself [01:18:36] yeah ok [01:18:59] andrewbogott: where do I find the new password? [01:19:34] should all be the same, just change ldap-eqiad to ldap-labs.eqiad and ldap-codfw to ldap-labs.codfw [01:19:47] ah ok [01:20:06] andrewbogott: ldap-labs.eqiad.wikimedia.org? [01:20:08] or .eqiad.wmnet? [01:20:16] moritz aliased the the old server names but of course the cert check doesn’t like that [01:20:24] ldap-labs.eqiad.wikimedia.org [01:20:32] just add -labs. to the existing names :) [01:22:37] andrewbogott: but existingone is ldap-eqiad.wikimedia.org [01:22:40] a dash not . [01:22:42] * YuviPanda tries [01:22:51] YuviPanda: i'm seeing this error on wikimetrics staging instance in labs - sudo: ldap_start_tls_s(): Connect error something going on? [01:22:58] oh i see ldap things on top [01:23:03] s/ldap-eqiad.wikimedia.org/ldap-labs.eqiad.wikimedia.org/g [01:23:06] andrewbogott: ok! [01:23:23] Krenair: try deployment-salt now [01:23:38] madhuvishy: probably the same change for you. Or better yet update your local puppet repo [01:23:41] I'm in [01:23:56] andrewbogott: okay [01:24:14] But I get the same error as madhuvishy [01:24:17] when trying to sudo [01:24:18] madhuvishy: sorry, I emailed the list about this but I think my email was confusing [01:24:37] and a password prompt [01:24:49] hm, sudo’s ldap server might be configured someplace else [01:24:58] andrewbogott: hmmm but i cannot git pull [01:25:03] uri ldap://ldap-eqiad.wikimedia.org:389 ldap://ldap-codfw.wikimedia.org:389 [01:25:03] andrewbogott: chasemp we need to get https://gerrit.wikimedia.org/r/#/c/222857/ back in somehow. allows people to add their own root keys... [01:25:56] madhuvishy: what instance and project? [01:26:04] i will get out of your way - let me know when you're done fixing the stuff you're already working on, my stuff can wait. [01:26:15] (wikimetrics-staging1.analytics) [01:26:34] Krenair: do you want me to fix any other project? [01:26:48] I don't think this is unbroken fully yet YuviPanda [01:27:23] Krenair: did you switch the uri in ldap.conf too? [01:27:51] I can't, YuviPanda [01:27:56] You need root to do that [01:28:00] I can't sudo [01:28:00] oh [01:28:03] haha [01:28:05] ofco [01:28:17] I can't log in as root myself because puppet is broken, despite the key being in place on deployment-puppetmaster [01:28:25] right [01:28:27] let me do it [01:28:30] basically [01:28:35] everything is broken [01:28:40] what else is new [01:28:52] I bet puppet's been broken for weeks :) [01:28:56] and people notice when things like this happen [01:29:02] madhuvishy: you have a local patch that can’t be merged with upstream. May I set it aside for now? [01:29:20] andrewbogott: hmmm, i'd have to ask milimetric [01:29:27] * andrewbogott does it anyway [01:30:15] yay, puppet works now...I bet that ldap change you did fixed it YuviPanda [01:30:22] oh, nope, wtf? [01:30:31] Krenair: really? that should be competletely unrelated to anything puppet touches [01:30:32] How did it update the root keys with my exact text and then fail a manual run? [01:31:18] Krenair: it reverted it back [01:31:23] to previous ldap.yamlo settings [01:31:29] urgh [01:33:22] Krenair: boooom http://tools.wmflabs.org/watroles/role/puppet::self::geoip [01:33:30] why what AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA [01:33:33] why would someone do that [01:34:39] YuviPanda: mine’s more broken I bet [01:36:13] andrewbogott: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=abb50762-93d0-4c9d-8853-adbbb6b56e00&project=deployment-prep®ion=eqiad [01:36:16] can't configure pages too [01:37:24] Krenair: I removed it with ldapvi on terbium [01:37:35] hm, that’s interesting [01:37:44] YuviPanda: are those pages broken for everything? Or just that one instance? [01:37:52] andrewbogott: not sure. that's the first one I checked [01:37:56] Krenair: ok, should work now. [01:37:59] It's broken for a bunch of hosts [01:37:59] Krenair: puppet run updated everything [01:38:00] maybe all [01:38:12] someone else was complaining about it on wikidata-query ones earlier [01:38:14] also WHY WOULD SOMEONE INSTALL PUPPET::SELF:GEOIP ONTO A SALT MASTER AAAAA [01:38:43] Krenair: can you check deployment-saltmaster now? [01:39:32] madhuvishy: any idea where passwords::wikimetrics is supposed to come from? [01:39:38] Notice: /Stage[main]/Exim4/Service[exim4]/ensure: ensure changed 'stopped' to 'running' [01:39:39] It’s referenced in puppet but not defined anyplace. [01:39:47] Is... that supposed to be present on deployment-salt? [01:39:51] it completed successfully YuviPanda [01:39:51] "# passwords::wikimetrics is not a real class checked in to any repository." [01:39:56] how was that supposed to ever do anything? [01:40:10] andrewbogott: ha ha i have no idea [01:40:29] andrewbogott: I'm pretty sure that it comes from the commit you removed :D [01:40:33] andrewbogott, YuviPanda: Dec 08 23:15:27 interestingly enough https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=6978ff87-d776-45fd-9a2e-4412d3926add&project=wikidata-query®ion=eqiad claims that host (wdq-beta) does not exist [01:40:41] wikimetrics and limn are done in a way where puppet will fail if you uncherrypick patches [01:40:48] YuviPanda: except shouldn’t my removed commit have also removed the /reference/ to it? [01:40:48] Dec 08 23:15:32 also https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=8978fa1c-c37a-4b69-bef8-6bf049d56aac&project=wikidata-query®ion=eqiad for db01 [01:40:49] etc. [01:40:50] I've told milimetric that if they die they die and we can not do much [01:40:56] andrewbogott: no, the reference is in ops/puppet :D [01:41:11] * andrewbogott is getting a bit stabby [01:41:13] I think at least [01:41:30] andrewbogott: IMO, we can't be held responsible for those 3 instances. they've been warned enough times. [01:41:39] yeah that still looks broken [01:41:45] we can probably add root keys and let analytics figure it out. [01:42:10] yeah, the configure issue seems more important anyway :| [01:42:23] Krenair: any other deployment-prep instance that needs rescuing? [01:42:28] andrewbogott: I'm sorry - there's a task to move away from self hosted puppet master - but it's super low priority and we've not gotten to it. but yeah, if i can get sudo back, we'll figure it out [01:42:37] I'm not aware of any others yet YuviPanda [01:42:41] Krenair: ok! [01:42:45] thanks [01:43:14] I'm going to try and fix the labs/private patch I showed earlier, making it easier for people to add their own root keys to their projects. [01:43:48] (03PS1) 10Yuvipanda: Revert "Revert "Allow addition of more root keys via hiera"" [labs/private] - 10https://gerrit.wikimedia.org/r/257808 [01:46:02] I can't log into labs bastion any more - known issue with ldap or the like? [01:46:03] madhuvishy: “My infrastructure could become unreachable at any minute” should not be a super low priority [01:46:26] andrewbogott: well we care very little about wikimetrics itself [01:46:29] andrewbogott: madhuvishy every time we change labs stuff one of us spends a few hours on those in stances, so you're basically just externalizing costs to us :( [01:46:37] madhuvishy: I can delete the project for you in that case [01:46:39] it’s easy! [01:46:49] andrewbogott: well, i wish [01:46:59] andrewbogott: madhuvishy so I think I can declare that this is the last time we'll help, since we've been saying this for years :) and if it's that super low priority I guess it'll have to die. [01:47:12] gwicke: which bastion? [01:47:26] andrewbogott: ugh, DNS down? [01:47:28] man watching this puppet apply is like a walk down memory lane [01:47:38] andrewbogott: no, no, not DNS down, I just mistyped [01:47:40] no panicking [01:47:43] YuviPanda: yeah that makes sense. [01:48:07] madhuvishy: where's the task so I can declare that on the task? [01:48:36] YuviPanda: i'll make one. also this can wait. i'll check with milimetric too. [01:48:37] gwicke I see you have successfully logged in? [01:49:11] https://phabricator.wikimedia.org/T101763 [01:49:13] madhuvishy: ^ [01:49:16] nevermind, I just fixed it on my end [01:49:54] had changed keys, but somehow my config didn't pick up the right one [01:50:25] sorry for the noise ;/ [01:50:41] madhuvishy: https://phabricator.wikimedia.org/T101763#1864407 [01:51:13] gwicke: np :) [01:53:31] YuviPanda: ah this. okay, thanks. should i make one to fix this? [01:53:43] madhuvishy: what doyou mean by 'this'? :) [01:54:35] 6Labs: Give madhuvishy and milimetric root on limn and wikimetrics instances - https://phabricator.wikimedia.org/T120900#1864414 (10yuvipanda) 3NEW [01:54:36] madhuvishy: ^ [01:54:40] YuviPanda: ah i thought i still couldn't sudo [01:54:44] i think i can now [01:54:50] madhuvishy: \o/ ok. [01:55:12] madhuvishy: your /var/lib/git/operations/puppet is in a horrible state [01:55:17] but for a few minutes you should have access [01:55:26] I dont’ know what to make of all these submodule conflicts [01:55:31] until puppet reruns hmmm [01:57:29] madhuvishy: I mashed everything into your ‘do not merge’ patch [01:57:42] but that means that patch is probably even harder to reconcile with prod than before [01:58:13] andrewbogott: okay [01:58:23] git reflog probably has the older version :D [01:58:27] * YuviPanda <3s git reflog [01:58:32] andrewbogott: what do you think of the wikitech issue? [01:59:11] I’m not sure why that would break. I’ll have a look [01:59:32] andrewbogott: thanks! [02:00:11] YuviPanda: how recently have you seen that work? [02:01:25] (03PS2) 10Yuvipanda: Revert "Revert "Allow addition of more root keys via hiera"" [labs/private] - 10https://gerrit.wikimedia.org/r/257808 [02:01:25] andrewbogott: hmm, yesterday? [02:08:12] 6Labs: Give madhuvishy and milimetric root on limn and wikimetrics instances - https://phabricator.wikimedia.org/T120900#1864429 (10yuvipanda) a:3yuvipanda [02:11:51] YuviPanda: does it work anywhere? or nowhere? [02:14:26] andrewbogott: so if you open up novainstance for deployment-prep [02:14:35] andrewbogott: the latest two instances (kafka03 and 04) created today work [02:14:39] all the old ones do not [02:14:44] ok [02:14:45] so I suspect it works for all new ones and none of the old ones [02:14:56] I’m tempted to wipe memcached [02:15:25] yeah [02:15:38] andrewbogott: have we done that before? [02:15:45] not today [02:15:51] andrewbogott: no I mean, ever? [02:15:56] sure [02:15:59] ah cool [02:16:02] it just breaks wikitech sessions [02:16:03] I wasn't sure how mc would react [02:16:08] yeah but that's ok [02:16:27] It’s a stab in the dark though [02:17:03] all of life is stabs in darks [02:17:43] didn’t help [02:18:48] andrewbogott: on NovaInstance, all of the ones without a working configure also have a missing link to the nova resource page... [02:18:57] where do those come from? [02:21:17] the nova resource pages are made by the OpenStackManager extension and updated when you change stuff about the instance [02:21:20] yeah, I see that too it’s surely related [02:21:32] what’s weird is that I see the nova call to verify the instance succeeding right before it declares no such instance [02:34:04] YuviPanda: can we update php on exec nodes? [02:34:17] liangent: to what version? [02:34:21] the trusty nodes already have 5.5 [02:34:27] YuviPanda: to match login host at least [02:34:35] liangent: yes, if you do '-l release=trusty' they match [02:34:46] liangent: that isn't default because too manyt ools rely on default being release=precise :( [02:35:25] YuviPanda: what's the breakage they got? [02:35:45] liangent: tools that don't work with 5.5 [02:35:56] liangent: I think it removes some ancient mysql interfaces... [02:37:11] exactly the opposite case than mine [02:37:27] after updating some libs I found they no longer working in 5.3 [02:37:33] for using $this inside closures [02:39:10] YuviPanda: another issue: job 166594 stuck with error message “can't get password entry for user "tools.liangent-php". Either the user does not exist or NIS error!” [02:39:24] YuviPanda / andrewbogott / madhuvishy: ok, I'm back, but I'm a bit lost about the above [02:39:29] where we left this is: wikimetrics is working fine [02:39:55] limn1 has a problem due to something that's actually broken in puppet, not related to our self-hosting at all [02:40:16] and it's not fair to say *every* time anything changes you spend *hours* on *each* of these instances [02:40:18] that's a bit dramatic [02:40:25] for the most part, I'm the one who spends those hours [02:40:37] I do ask for help here and there, and I appreciate the help [02:40:46] but it's certainly not hours [02:41:18] and most of the time not related to our custom puppet code on there, but to breaking changes in puppet [02:41:33] so, summary: everything's fine, nobody needs to pay attention to these instances [02:42:04] except puppet continues to be broken on limn1 and I'd love someone to fix that at some point. If it's not fixed, the next time I need to sudo on that machine I'll just migrate everything to a new box instead [02:42:22] but that's this error: "Error 400 on SERVER: Could not find resource 'Exec[compile puppet.conf]' for relationship on 'Class[Puppetmaster::Ssl]' on node limn1.analytics.eqiad.wmflabs" and nothing we're doing [02:45:12] milimetric: when users ask us to help with access because they can’t ssh… should I not help and tell them ‘everything is fine’? [02:46:29] andrewbogott: yeah, in this case, because we know the risks associated with limn1 and wikimetrics1, I'd rather you tell us we're screwed instead of taking on extra work that you didn't sign up for [02:46:46] the way this is set up is my fault (well, not really, but I inherited the puppet setup) [02:47:08] that doesn’t sound the same as ‘everything is fine’ but ok :) [02:47:54] everything is "normal" i guess [02:48:59] milimetric: 'breaking changes in puppet' - we expect puppet to run and not have months of catching up to do. [02:49:19] milimetric: 'the next time I need to sudo no that machine I will migrate everything to a new box' sounds good to me :) [02:49:43] milimetric: I filed that task after the last time I helped look at it. I agree that you're the one spending most of that time, but you *are* blocked on us every time it breaks badly. [02:50:26] YuviPanda: yeah, but that's technical debt that my managers didn't choose to pay down, so I'm not worried about it [02:50:32] if they want to prioritize it, they will [02:51:04] YuviPanda: but puppet should be enabled to run automatically and just git pull --rebase on those boxes [02:51:08] milimetric: it is. [02:51:11] I login every once in a while to make sure that's happening [02:51:31] right, so it's only when there are breaking changes, but I never let it slip for months like that [02:51:37] milimetric: and the patches on top break rebases now and then and when andrewbogott got puppet to run again today it was applying a *very long list of chagnes* from a long tiem ago. [02:51:37] 'cause I see the shinken alerts [02:51:51] hm [02:52:06] ok, so we can do something different then [02:52:17] anyway, I think we all agree here. If analytics considers this super low priority, it's even lesser of a priority for us and we should explicitly ignore issues caused by puppet breakage in this kind of situation. [02:52:19] we can just shut down limn1 and wikimetrics1 and tell people what's wrong so they can pay down the debt [02:52:27] it would kill all dashboards, piwik on labs, and wikimetrics [02:52:38] indeed, and then maybe it won't be low priority :) [02:52:52] or maybe they really are low priority and nobody would miss them [02:52:54] right :) I personally think it'll become medium - high pretty quickly :) [02:53:09] indeed, and if I were you that's what I would have done but you are a nicer person than me :D [02:53:43] ok, thx for the help, but don't spend any more time on it. I'll send a notice that this is what we're planning on doing [02:54:03] milimetric: I've filed a task to make sure additional people can get added to the root key on projects in the analytics project, and then I can consider ourselves truly out of your way and then you can deal with it as you see fit :) [02:54:10] and this is useful for other projects too anywya [02:55:45] (03PS3) 10Yuvipanda: Revert "Revert "Allow addition of more root keys via hiera"" [labs/private] - 10https://gerrit.wikimedia.org/r/257808 [02:55:55] I think ^ will work [02:57:30] 6Labs: ldap queries failing when looking up domain entries - https://phabricator.wikimedia.org/T120904#1864497 (10Andrew) 3NEW a:3Andrew [03:00:24] (03CR) 10Yuvipanda: [C: 032 V: 032] "Let' see how this goes!" [labs/private] - 10https://gerrit.wikimedia.org/r/257808 (owner: 10Yuvipanda) [03:01:55] Krenair: hey! want to help test ^? [03:01:59] ok [03:02:03] Krenair: we should add your key to root on deployment-prep [03:02:38] YuviPanda: Tsk, Krenair has more than enough to fix in VE right now. ;-) [03:07:01] James_F: tch tch :P [03:07:16] * James_F grins. Go for it. Krenair is awesome. [03:07:25] :) [03:07:35] Krenair: in https://wikitech.wikimedia.org/wiki/Hiera:Tools [03:07:40] you can see how valhallasw has his key added [03:08:06] yeah, I did something similar in deployment-prep [03:08:28] Krenair: you made it a list [03:08:31] Krenair: it should be a dictionary I think [03:08:39] Krenair: with the keys being your name and the value being the key [03:08:51] oh but it's supposed to be a hash [03:08:52] right [03:09:32] (03PS1) 10Yuvipanda: Move template to right place [labs/private] - 10https://gerrit.wikimedia.org/r/257819 [03:09:45] (03CR) 10Yuvipanda: [C: 032 V: 032] Move template to right place [labs/private] - 10https://gerrit.wikimedia.org/r/257819 (owner: 10Yuvipanda) [03:10:38] oops: "Could not retrieve catalog from remote server: Error 400 on SERVER: values(): Requires hash to work with" [03:11:08] heh [03:11:46] https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=222055&oldid=222051 [03:11:55] shouldn't that have fixed it? [03:13:30] Krenair: is that one straight line broken up by mw rather than two lines? [03:13:36] Krenair: also hiera is cached for like 30s or something [03:14:32] YuviPanda, done, it worked [03:14:44] it added my key to /etc/ssh/userkeys/root, login works [03:14:48] Krenair: \o/ awesome [03:15:28] Krenair: can you email releng@ or qa@ (or whatever ml is responsible for betacluster) to add more people to that list so they can login in cases of ldap / ssh failure? [03:15:31] that's much easier than having to local hack it in on the deployment-prep puppetmaster [03:15:35] yeah [03:15:38] definitely [03:17:02] releng@ is the private team list which I'm not on [03:17:52] yep, it's qa@ according to https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep [03:19:35] 6Labs: ldap queries failing when looking up domain entries - https://phabricator.wikimedia.org/T120904#1864512 (10Andrew) Here is the (or at least one) record that we're failing to load: # eqiad, hosts, wikimedia.org dn: dc=eqiad,ou=hosts,dc=wikimedia,dc=org objectClass: domainrelatedobject objectClass: dnsdoma... [03:20:18] 6Labs: ldap queries failing when looking up domain entries - https://phabricator.wikimedia.org/T120904#1864513 (10Andrew) I suspect that wikitech is acl'd out of everything under ou=hosts,dc=wikimedia,dc=org [03:20:41] andrewbogott: would that explain it working for new instances? [03:20:54] I don’t know [03:21:58] It really should fail for everything since it can’t load eqiad.wmflabs [03:23:14] YuviPanda: I don’t see ‘instance id’ for anything in deployment-prep [03:23:22] tell me again where it’s working? [03:23:26] andrewbogott: kafka03 and 04 [03:23:41] what project is that? [03:24:14] andrewbogott: deployment-prep [03:24:19] andrewbogott: in deployment-kafka03 and 04 [03:24:23] to be more exact :) [03:24:30] andrewbogott: I don't see them now [03:24:39] andrewbogott: i wonder if the memcached clear cleared them out [03:24:41] yeah maybe they were cached somehow [03:24:43] probably [03:25:01] So I think I’m going to dump this on Moritz and go to bed. I don’t much know how to debug the server-side stuff [03:25:11] that seems fair [03:25:36] 6Labs: ldap queries failing when looking up domain entries - https://phabricator.wikimedia.org/T120904#1864517 (10Andrew) a:5Andrew>3MoritzMuehlenhoff [03:25:50] I don’t like it being broken but no one is screaming at the moment [03:28:38] yeah. [03:28:41] YuviPanda, oh, damn. Since I'm not subscribed it's been held for moderator approval [03:28:52] Krenair: 'tis ok, I'm sure someone will see it soon enogh [03:28:57] Krenair: <3 thanks for sending that out [03:29:07] greg-g, zeljkof ^ [04:03:50] Krenair: approved [04:04:07] ty [04:04:27] YuviPanda, https://lists.wikimedia.org/pipermail/qa/2015-December/002452.html [04:33:43] someone is killing NFS [04:40:05] not anymore, maybe [07:35:57] hey, I just made a new service group but it doesn't have replica.my.cnf [07:36:07] and I can't make databases [07:36:17] YuviPanda: Coren ^ [07:38:11] it's there now [07:38:13] :) [07:40:08] Amir1: how long did you wait? [07:40:20] YuviPanda: about ten min. [07:40:31] I restarted it for good measure [07:40:36] it might be running in 10min intervals anyway [07:40:59] thanks [07:41:27] I'll wait some more next time :) [07:41:40] ^ that's the motto of labs, right there :) [07:41:55] btw have you seen this? http://tools.wmflabs.org/wd-analyst/index.php I made it with your help [07:41:57] thanks [07:42:03] :))))) [07:42:49] \o/ [07:42:51] awsesome :D [07:42:54] * YuviPanda smiles very broadly :D [07:43:34] :)))) [07:54:08] 6Labs: ldap queries failing when looking up domain entries - https://phabricator.wikimedia.org/T120904#1864690 (10yuvipanda) p:5Triage>3Unbreak! Increasing priority since this makes new instance usage unusable. [08:12:53] (03PS2) 10Polybuildr: Add .npm-debug.log* to .gitignore [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/241919 [08:15:19] (03CR) 10Yuvipanda: [C: 032] Add .npm-debug.log* to .gitignore [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/241919 (owner: 10Polybuildr) [08:15:53] (03PS2) 10Yuvipanda: Add connections.yaml to .gitignore [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/241921 (owner: 10Polybuildr) [08:16:02] (03CR) 10Yuvipanda: [C: 032 V: 032] Add connections.yaml to .gitignore [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/241921 (owner: 10Polybuildr) [09:50:28] (03PS1) 10Alexandros Kosiaris: Remove an extra passwords::ldap::corp [labs/private] - 10https://gerrit.wikimedia.org/r/257846 [09:51:28] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Remove an extra passwords::ldap::corp [labs/private] - 10https://gerrit.wikimedia.org/r/257846 (owner: 10Alexandros Kosiaris) [10:04:11] 6Labs, 10DBA, 5Patch-For-Review: watchlist table not available on labs - https://phabricator.wikimedia.org/T59617#1864853 (10jcrespo) Because of the size and number of tables, this will take me some time (and it will require disrupting labs lag), but I will put it on my backlog and attend it as soon as I can. [10:32:10] 6Labs, 6TCB-Team, 7Tracking: Create labs projects for lizenzhinweisgenerator - https://phabricator.wikimedia.org/T120925#1864901 (10Addshore) 3NEW [10:36:35] Amir1: ! [10:37:29] Just wondering for http://tools.wmflabs.org/wd-analyst/index.php are all of the number pre computed and in some table somewhere? [10:37:38] or are they grabbed on the fly? [10:38:36] addshore: you can read the source :-) the table you're looking for is probably "s52781__wd_p" [10:38:53] * goes to read the code* [10:39:08] (/data/project/wd-analyst/public_html) [10:39:24] firs thing I notice if you are using some python dumps reader, you should use a php one, and then you can sue the php datamodel objects etc ;) [10:40:22] we should have a way to compile a php datamodel to python! [10:40:33] valhallasw: sofixit? :P [10:40:53] it shoudlnt be that hard right? [10:41:10] parse php to ast, convert ast to python ast, dump as python [10:41:20] the last is actually not even completely necessary [10:41:58]