[06:33:26] !log account-creation-assistance running apt-get updates and reboot accounts-application [06:33:28] Logged the message, Master [06:45:38] Anyone around? I'm getting "read-only filesystem" errors on the account-creation-assistance-home gluster [08:36:13] Coren: or anybody: a user complains that there is no phpMyAdmin on labs. is it installed? i don't do much php so i wouldn't know. [10:44:54] FastLizard4|zZzZ: sorry I missed you… I've restarted that volume, please let me know if you continue to have trouble. [10:59:43] paravoid: a moment of your time? [11:00:10] yes? [11:01:47] I've been upgrading the puppet repos on a few neglected puppetmaster::self instances. [11:01:58] And on severaly of them… once things are up to date, I can no longer log in. [11:02:09] My existing session persists, and other users seem unaffected. [11:02:33] for instance, my latest victim, 'faidon-test' in the puppet project. [11:02:53] heh :) [11:03:03] My questions are… 1) can you still ssh in to that instance? [11:03:15] 2) if so, can you see any reason why it won't let me in anymore? [11:03:50] are you trying to login as root? [11:04:42] I managed to login as "faidon", not as root thoug [11:04:51] Ah, so your root key is broken too? [11:04:53] For me it's both [11:06:17] root@faidon-test:~# grep userkeys /etc/ssh/sshd_config [11:06:17] root@faidon-test:~# [11:06:34] so /etc/ssh/userkeys/root/.ssh/authorized_keys isn't being used [11:06:42] but /root/.ssh/authorized_keys is used instead [11:06:49] which is empty, since the instance was new enough [11:08:00] ok, so that would account for the root key problem but not the 'andrew' key problem... [11:08:06] * andrewbogott digs through git [11:08:15] is the spanish wikipedia still not replicated, or am i doing it wrong? [11:09:35] paravoid, haven't found it yet, but I suspect this patch: https://gerrit.wikimedia.org/r/#/c/90098/ [11:10:54] ok, i was doing it wrong. it moved to a different server and i didn't update /etc/hosts on my vm yet... [11:11:06] andrewbogott: yeah, that's what I was thinking too [11:13:28] heh, $::realm is production [11:14:09] whoah, really? [11:14:16] That would do it! [11:14:29] at least in the ssh module [11:14:35] where do we define $realm... [11:15:20] this feels familiar, like I had an order of operations thing between base and realm a few weeks ago. Thought I fixed it though... [11:15:22] was it ldap? [11:15:36] so, maybe realm is defined after the ssh module is loaded [11:16:16] It's defined in ldap puppetVar: realm=labs [11:17:08] indeed it is [11:18:59] It was $realm before the refactor, $::realm after [11:20:59] that makes no difference, fortunately [11:26:09] so, wait [11:26:09] so presumably realm.pp is getting loaded before that var is pulled in from ldap [11:26:15] …waiting… [11:26:17] is it only on puppetmaster::self instances? [11:26:26] is this happening [11:26:40] I think so. And only ones that are oldish. [11:28:31] are things loaded in a different order for ::self instances? [11:28:46] that would be weird [11:29:43] it looks to me like $realm is correct on puppet-testing-6 [11:30:03] which also has a self-hosted puppetmaster, but is a younger instance [11:32:14] huh [11:32:19] puppet.conf is very weird [11:32:24] has [main] and [agent] twice [11:32:53] I have been hand-editing that file a bit, to add modulepath. I didn't duplicate any sections though [11:33:18] let's see [11:36:08] you're regenerating that file? [11:38:41] /etc/puppet/puppet.conf.d/10-main.conf gets created [11:39:07] that's happening because puppetClass: puppetmaster::self is also ignored [11:39:14] the whole puppet/LDAP integration is broken [11:39:48] Ah, it's a circular problem, ldap is broken which means… ldap is broken [11:39:52] no [11:39:56] no? [11:39:57] not sure why ldap is broken yet [11:40:16] it wasn't the two sections in puppet.conf, I stripped those and it was still broken [11:41:01] y'think if I copy over a known good puppet.conf? Mind if I try that? [11:41:06] Or is that essentially what you just did? [11:41:31] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class passwords::puppet::database for faidon-test.pmtpa.wmflabs at /etc/puppet/modules/base/manifests/init.pp:67 on node faidon-test.pmtpa.wmflabs [11:41:35] uhm [11:41:36] that's new? [11:42:00] did you do something? [11:42:01] That's because it's still using an old puppet.conf that doesn't have the modulepath set [11:43:10] So it's still reverting to a pre-modularized puppet.conf. Somehow some puppet.conf settings from before my upgrade attempts are sticky. [11:44:57] and/or you removed the section that included the modulepath def [11:45:12] it's a little more complicated than that, but fixed [11:45:46] (there's a puppet.conf.d that gets concatenated into puppet.conf whenever it gets changed, so me removing 10-main.conf made it regenerate it, and 10-self was the pre-modulepath version) [11:46:41] so, for future instances... [11:46:53] just wiping out puppet.conf.d should get things unstuck? [11:47:18] I still haven't figured out why ldap is broken [11:47:32] (it still is) [12:14:03] paravoid, still messing with that instance? (I'm not, in order to avoid toe-stepping) [12:14:51] sorry, not anymore [12:14:55] not feeling too great atm [12:15:40] no worries -- feel better! [12:16:05] btw, is that instance still of interest to you? Or should I delete it if/when I solve this problem? [12:36:34] I've used in the past as generic personal testing box [12:36:57] I haven't used it in a while but I'd like to keep having it, although I don't care about it running puppetmaster::self anymore [13:48:21] Webserver is busy OR angry OR down [14:04:24] hedonil: Works for me; what are you having issues with? [14:05:24] JohannesK_WMDE: There is no phpMyAdmin on labs, and there will never be one. If the user wants to talk to a database, they should use a client on their own computer. [14:07:13] intermediately slow as heck [14:08:37] Ah, we're still getting hammered by SYNs on HTTP. [14:08:52] Coren:bad guys [14:10:07] Doesn't look like; it's looking more as though we're getting too many geohacks for geohack to handle. Perhaps something is crawling a project. [14:10:14] geohack should really use the new scheme. [14:10:45] So it's more of an "accidental" DDOS than a malicious one. [14:14:48] Coren: If it's the same tool as geohack on toolserver - it's also slow as heck right now. http://toolserver.org/~geohack/ [14:15:03] It's the same tool, yeah. [14:15:21] It's definitely being hammered on by something. [14:35:13] Coren / andrewbogott - do you guys want to investigate those broken instances again? [14:40:26] MaxSem: If you've got at least one which still allows root login, sure. [14:40:54] Coren, I can sacrifice mobile-osm2 for that:P [14:43:09] MaxSem: Allright, lemme dig in. [14:44:54] MaxSem: puppet files to apply at all there "Could not find class role::osm::db for mobile-osm2.pmtpa.wmflabs" [14:44:59] fails* [14:49:31] Coren, fixed [14:51:13] err: Failed to apply catalog: Could not find dependent Package[sudo-ldap] for User[root] at /etc/puppet/private/manifests/passwords.pp:6 [14:51:19] Bleh. [14:52:22] funny - it worked once for me [14:52:47] anyway, do you guys still need those instances for dissection? [14:53:33] Hm, were they puppetmaster::self previously? [14:53:48] yup [14:54:01] that's why Andrew updated them [14:55:25] Yeah, I'm thinking this is partially broken puppet config; that's generally hard to salvage but not very useful for investigative purposes. I don't think there's anything valuable for us on them, except the "if an instance has been too long without its puppet client config updated it'll probably break" datapoint we already expected. :-) [14:56:58] and you still have 20 instances to break:P [15:04:45] MaxSem, Coren, sorry was sleeping... [15:04:56] But, overnight faidon helped sort out some of the issues. [15:05:28] That is, we sorted out the aforementioned puppet problem. But keys are still broken :) [15:06:43] The problem is interesting… classes from puppet modules are getting applied before realm=labs is applied from ldap [15:07:56] Oh, eew [15:08:24] so it expects your prod keys? [15:09:21] I think puppet/ldap integration is completely broken [15:09:23] for those [15:09:25] not just the realm [15:09:33] classes don't get applied either [15:09:53] so can these instances be ditched now? [15:10:26] I'm interested in finding out why [15:11:02] Yeah, it's broken in such a way that… I don't feel confident it won't suddenly break in the same way on other instances [15:11:53] Yeah, I was about to say that if it was as simple as the wrong realm then our prod keys should have been deployed. [15:12:52] $::realm is definitely "production" [15:12:57] but I checked Andrew's keys yesterday - they were the 2 he expected [15:13:36] Here is why I'm special: I have a different username in prod and labs due to andrew gerrit having dibs on the name 'andrew' in production. [15:13:53] So probably there are root keys for the name 'andrew' and probably they are his, not mine. [15:13:58] on account of it being production instead of labs :( [15:14:03] *shrug* or something [15:14:08] oh heh [15:14:19] but no [15:14:23] Although, wait, I guess everones' root keys are broken not just mine [15:14:37] they were your keys - you told me their names, remeber? [15:14:43] bah. [15:15:09] Anyway, specific symptoms are not necessarily interesting since the instance should definitely not think it is in realm 'production'! [15:15:17] root keys are broken because the userkeys sshd_config stanza isn't getting deployed [15:21:00] paravoid: When I looked at this before, I fretted a lot over that "if $::realm == undef" line because it is clearly relient on order of operations, /and/ it gets traversed during 'import' rather than 'include' which seems unreliable. [15:21:28] puppetmaster::self isn't in the defined classes, so puppetClass gets ignored too [15:21:35] so it's more than just realm [15:21:56] Oh, you mean that ldap settings aren't getting loaded at all? [15:22:01] Not just loaded out of order? [15:23:00] I think so [15:23:52] So that points the blame back at puppet.conf [15:25:40] --- /etc/wikimedia-realm 2013-11-01 10:51:13.944714344 +0000 [15:25:40] +++ /tmp/puppet-file20131101-15583-zk02kp-0 2013-11-01 15:25:30.019064922 +0000 [15:25:43] @@ -1 +1 @@ [15:25:46] -production [15:25:48] +labs [15:26:04] fun [15:26:07] so, yes [15:26:19] restarted puppetmaster, fixed now [15:26:25] after having changed puppet.conf [15:27:24] what did you change about puppet.conf? [15:30:51] basically rm /etc/puppet/puppet.conf.d/10-main*; cp /etc/puppet/puppet.conf.d/10-self* /etc/puppet/puppet.conf [15:31:26] ah, ok. [15:31:48] MaxSem, want to try that on mobile-solr2? [15:32:11] sure [15:33:09] don't forget to /etc/init.d/puppetmaster restart [15:33:24] not sure if the puppet.conf change actually made the difference [15:33:35] or if just a puppetmaster restart alone would do it [15:34:02] oh… MaxSem, if it's not too late, try just a puppetmaster restart first? [15:34:14] too late:P [15:34:32] se have solr3 to try though;) [15:34:42] yep! [15:35:07] mobile-solr2 still doesn't love me [15:35:35] wait, about to force a run [15:35:47] 'k [15:35:50] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class passwords::puppet::database for mobile-solr2.pmtpa.wmflabs at /etc/puppet/modules/base/manifests/init.pp:67 on node mobile-solr2.pmtpa.wmflabs [15:37:45] in puppet.conf, after templatedir add 'modulepath = /etc/puppet/private/modules:/etc/puppet/modules' [15:38:11] how does tool labs account creation/ tool creation work these days? [15:38:50] for dominic's benefit: 01 15:38:11 < jeremyb> how does tool labs account creation/ tool creation work these days? [15:39:13] Dmcdevit: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools [15:39:17] see "getting access" [15:40:16] Thanks. [15:40:24] Dmcdevit: do you just have a wikitech account or have you ever tried logging in to one of the servers by ssh? [15:40:55] And I don't even know how to log into my wikitech account now. What's this token field? :-) [15:41:06] hah, ignore that [15:41:07] I literally did nothing but register the account. [15:41:18] ok, then you'd need to set up keys even [15:41:21] so, later [15:41:31] you want csv? [15:41:58] andrewbogott, try now [15:42:19] jeremyb: Sure, that works. [15:42:23] MaxSem: nope, same :( [15:43:40] funny thing, after I edited puppet.conf it first choked on nagios-nrpe-server but then started complaining about passwords::puppet::database [15:43:43] again [15:43:47] Dmcdevit: anyway, make an ssh key and register the public half in wikitech (in the wiki) [15:44:18] MaxSem, yeah, that's because the latent files in puppet.conf.d are returning to haunt you. [15:45:07] So probably you should follow paravoid suggestion but also set the files in that dir aside [15:46:13] nah, ran rm /etc/puppet/puppet.conf.d/10-main* && cp /etc/puppet/puppet.conf.d/10-self* /etc/puppet/puppet.conf [15:46:29] then restarted puppetmaster [15:46:41] then puppetd -tv barfs again [15:46:43] what's up with tools-login? idk if this is normal but just pulling up a man page is taking a while [15:46:55] i.e. longer than i took to write that about it taking a while [15:58:22] MaxSem: OK, desperate measures: Move both puppet.conf and puppet.conf.d out of the way. Then create a new puppet.conf with contents https://dpaste.de/uNwH [16:00:18] jeremyb: cron is also complaining [16:00:37] Coren, petan, could you check what's happening with tools-login [16:00:38] ? [16:01:07] Looking at it now. [16:01:19] Ah, some evil people are running bots on it. How rude. [16:01:28] * Coren kills. KIIIIILLL! [16:01:47] there's also a mysql quickly using up all memory [16:02:08] 31933 elgransc 20 0 1701m 1.5g 2200 S 21.5 78.8 0:20.85 mysql [16:02:16] (yes, that's 78.8% MEM) [16:02:36] hahaha [16:02:39] now stable around 80% [16:03:35] * Coren sighs. [16:03:37] Dmcdevit: please apply with the link at getting access (even if you haven't done the key thing yet) [16:03:47] I was *hoping* I wouldn't have to start putting ulimits on -login [16:04:09] jeremyb: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Dominic :-) [16:04:14] andrewbogott, done [16:04:25] Dmcdevit: i'll have to purge the page that uses that then [16:04:44] yes, after a purge you're there [16:04:53] MaxSem: do you get multiple clean puppet runs now? [16:05:19] clean as with just one tiny nrpe failure:P [16:05:32] Dmcdevit: i'm thinking maybe a nara tool? or maybe there's some existing collaborative glam stats tool? [16:05:50] Dmcdevit: tools are cheap and are a way that we can share access/responsibilities [16:06:07] and have stuff live in an appropriately named place instead of just sitting in my home :) [16:06:52] jeremyb: Magnus has a category already for his GLAM tools, if that's what you mean. [16:06:56] http://tools.wmflabs.org/glamtools/ [16:07:02] MaxSem: ok, root key works again! Woo [16:07:09] Want to do the same on solr3? [16:07:10] :) [16:07:16] Just the clean-and-paste part I mean. [16:07:33] sure [16:07:42] Dmcdevit: so, up to him then i guess [16:11:01] he just keeps restarting it... [16:11:15] !log tools petrb: restarted terminator daemon on -login to sort out memory issues caused by heavy mysql client by elbransco [16:11:15] How to make scheduled tasks using cron? [16:11:18] Logged the message, Master [16:11:31] hi [16:11:59] valhallasw: I restarted terminatord which was down for some reason (that logically makes the server almost unusable when it's down) [16:12:10] so now it should be all ok [16:12:18] Ahmad_Sammour: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Scheduling_jobs_at_regular_intervals_with_cron [16:12:21] local-dplbot is also running on -login [16:12:35] too many people are running insane tasks on login that should have never run there, terminator can take care of these :o [16:13:07] Yes, I'm afraid I'll have to tighten the screw some. :-( [16:13:43] jeremyb: It's currently running while attended by a maintaner; that's okay for short periods for debugging and such. [16:13:59] Coren: ok [16:14:01] well, it's not if it starts eating more than 2gb of ram :P [16:14:13] petan: Indeed. [16:14:40] that guy elbramsco is periodically starting mysql client which selects hundreds of thousands of lines from enwp [16:14:46] petan: with an n [16:15:07] andrewbogott, try mobile-solr3 [16:15:09] everytime it hit some 190 000mb of ram and then get killed and then again on different set of sql lines [16:15:26] MaxSem: yep, seems good. Thanks! [16:15:32] wee [16:15:34] petan: There's a shell script looping, and unattended since he didn't react to my messages. I just HUP'ed him [16:15:47] ok [16:15:59] so now you know how to avoid this when upgrading? [16:17:58] petan: I told you about warn-screen right? [16:18:11] yes I think [16:18:13] !toolsadmin [16:18:13] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin [16:18:33] I do this to send a message when killing stuff that lives in a screen. [16:18:35] but unless I inserted that to docs I already forgot how to use it [16:18:50] terminator is sending emails with explanations [16:19:01] That works too. [16:19:08] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin#If_they_run_them_in_a_screen [16:19:09] yes this [16:20:35] Coren, deleted edits? [16:20:53] Cyberpower678: Aren't you tracking the tickets? [16:22:48] Enough. [16:23:15] MaxSem: is mobile-osm2 still in use or can it be deleted? [16:23:39] nope, deleting a few my instances now [16:24:08] great, thanks [16:24:23] Coren, I am, but I'm confused. [16:24:35] "Successfully deleted instance i-00000507 (mobile-solr2), but failed to remove its DNS entry." [16:26:03] Cyberpower678: What confuses you? [16:26:15] !log mobile Deleted old instances mobile-solr2, mobile-solr3 and mobile-osm2 [16:26:17] Logged the message, Master [16:28:11] aude: can you access wikidata-testclient? And do you know if it's still in use? [16:28:35] They say it's ready, but I don't see it. [16:28:41] Coren, ^ [16:29:55] Where in 49088 or 49189 does it say it's ready? [16:37:52] andrewbogott: we are using everything, though some of our test instances need to be updated to derive from mediawiki-vagrant (i think) [16:38:15] is there an issue? [16:38:50] aude: I'm switching things from using puppetmaster::self to role::puppet::self. That change is a noop, but I've been going through and checking that those instances have good puppet runs before changing... [16:38:54] that instance I can't connect to at all. [16:39:04] i can connect [16:39:27] ok. Do you mind making that change, then? [16:39:31] ould not start Service[nagios-nrpe-server]: Execution of '/etc/init.d/nagios-nrpe-server start' returned 2: at /etc/puppet/modules/nrpe/manifests/init.pp:77 [16:39:41] what do i change? [16:39:56] Actually I can change it -- if you're getting that message then puppet is more-or-less happy :) [16:40:22] it's probably good for me to know what is changing [16:40:27] in case i have to change it in other places [16:40:40] It was just on this page here: [16:40:41] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=8cfbe5e1-95ab-47ce-ab02-8ca73dbd93fc&project=wikidata-dev®ion=pmtpa [16:40:43] that's if it is simple [16:40:55] I just unchecked puppetmaster::self and checked role::puppet::self [16:41:01] ok [16:41:03] Pretty soon the puppetmaster::self option will vanish from that page [16:41:08] Once it's purged from all labs instances. [16:41:11] (14 left!) [16:41:22] testclient is the one where i am experimenting and want to replace it with a multi wiki setup [16:41:25] for multiple clients [16:41:28] 'k [16:41:38] probably to use vagrant too :) [16:42:04] OK… we aren't actually ready for people to use labs vagrant because of legal issues… best to hold off on that if you can stand it. [16:42:54] I need to go meet people for lunch… leave me a page here if I turn out to have broken anything :/ [16:44:35] andrewbogott_afk: oh really? [16:44:47] can i experiment with it? [16:53:00] If i use cron to schedule tasks, does the tasks will work until the computer OFF? [16:53:36] i think you missed a word? [16:53:52] do you have some other deeper question you haven't asked yet? [17:00:28] any help? [17:07:03] Cron doesnt work in windows, right? [17:08:27] Ahmad_Sammour: why do you ask those questions here? they do not seem to have to do with labs at all [17:14:43] Ahmad_Sammour: it seems you've missed some basic understanding of the systems you're using. and of where your computer ends and another begins [17:16:18] we do have an encyclopedia you could consult [17:17:30] https://en.wikipedia.org/wiki/cron https://en.wikipedia.org/wiki/Secure_Shell https://en.wikipedia.org/wiki/Linux https://en.wikipedia.org/wiki/Shell_account [17:17:57] Ahmad_Sammour: have a look at those articles and then come back :) [17:18:23] Ahmad_Sammour: say something! [17:18:55] Yeah [17:21:11] Just one thing (yes or no), Does Cron run at windows? [17:21:37] Hi guys! I'd like to dump the content of the revision table in CSV format. Any idea? [17:21:42] Ahmad_Sammour: no [17:21:54] Ahmad_Sammour: and this channel is specific to a project. are you using wikimedia labs [17:21:54] ? [17:24:21] Ryan_Lane: Sorry, I don't understand what you mean. [17:26:12] I actually use wikimedia labs right now. [17:27:10] elgranscott: why? [17:27:30] elgranscott: Download a dump and process it on your own computer? [17:29:23] Coren|Dark, what is the actual status then? What are we waiting for? [17:29:38] anomie: yes [18:02:37] MrZ-man: I want the revision table to analyze some data for a PhD thesis [18:08:01] elgranscott: it's probably more efficient to run the analysis in SQL on the database [18:08:22] but if you think you really need a dump, please check http://dumps.wikimedia.org/backup-index.html [18:09:04] I think you want stub-meta-history.xml.gz [18:10:31] valhallasw: I need to analyze the data with a software I've developed ;) [18:18:38] elgranscott: it's fairly trivial to process the meta dump with some Python to create a CSV file or push it into a local database... [18:36:00] valhallasw: are all data of revision table in that dump file? [18:43:15] elgranscott: I think so, but I'm not sure. Download one from a smaller wiki to check [18:51:17] IIRC from processing it has all, and for those where content was deleted it is marked as deleted [19:22:54] aude: The legal issues aren't very substantial… just, we're trying to make sure that there's a very clear public distinction between labs and production wikis, and display the proper labs TOU and such. [19:23:04] wikimedia-singlenode puppet has been vetted for all that, labs-vagrant has not [19:25:15] aude: If you are still around… want to make switch maps-pgsql3 to role::puppet::self as well? [19:27:36] hi [19:29:28] hi! [19:31:56] i am not actively using that one so deleted it [19:32:14] can come back later with a fresh instance [19:32:33] any timeframe for labs-vagrant? [19:38:56] aude: Um… not really. I haven't looked at it much. [19:40:44] ok [19:43:08] aude, how about maps-pgsql3? [19:43:22] (That's the last one I think :) ) [19:44:16] deleted [19:44:23] so easy! [19:44:27] yeah! [19:44:37] * aude can start fresh when i come back to maps stuff [19:45:19] do you happen to know about maps-varnish? Looks like it might be crashed or oom or something [19:46:29] hm… I can't ssh into maps-tiles either [19:49:14] aude: And, more general question, why do so many instances in 'maps' use puppetmaster::self but don't actually have any changes to their puppet config? [19:49:32] don't know [19:49:38] 'k [19:49:42] how can you not ssh into them? [19:50:30] * andrewbogott tries again [19:51:30] maps-tiles1 is hanging when I ssh. Can you log in? [19:59:38] let me try [19:59:58] thanks [20:01:46] andrewbogott: can't ssh into that one [20:01:56] ok. [20:02:02] It's not personal, at least :) [20:02:02] i wonder if we still need it [20:02:32] its' the maps warper one and miniatlas that i would definitely not delete [20:02:46] maps-varnish, maps-tiles1 and maps-tiles2 are all exhibiting that same symptom [20:02:52] miniatlas is fine, I was just there [20:02:54] the others, not sure how actively they are being used [20:03:02] and maps-warper should be newer [20:03:23] OK. Well, maybe I'll just change the puppet conf for those three and not worry too much about logging in and getting clean runs. [20:04:07] i'd check with MaxSem about those [20:04:19] yep? [20:04:32] do we still need maps-tiles instances on the maps project? [20:04:43] * aude thinks they are not actively in use, but not sure [20:05:46] I think we don't need any of those instances at all, bblack is experimenting somewhere else, right? [20:06:47] i think so but no idea [20:08:25] i would also check with kolossos, although don't think he's doing anything on labs yet [20:08:46] 99% sure he is not using them [20:09:59] them not working seems like a good sign that no one is using them. [20:10:13] But, I made the puppet change so I'm satisfied. Y'all could delete them if you're sure they're obsolete. [20:11:54] and ok [20:11:58] andrewbogott: [20:13:39] petan: what can you tell me about the toolsbeta-mc instance? I can't log in... [20:43:55] YuviPanda: So… the proxy and api work feels totally stalled. I can /almost/ pick up the work entirely myself… is it important that any of your pending changes make it into the upcoming package? [20:45:46] andrewbogott: nope, it was just to make stuff easier for packagina [20:45:48] so feel free to self merge, even [20:45:51] sorry :( [20:46:00] I can spend time on it when I head back to India in a week [20:46:16] but until then looks like I'm trying to maximise f2f time in the office with folks [20:46:29] YuviPanda: OK… I think all I need to proceed then is advice about how to migrate data from proxy-dammit to a new host. That's just a couple of rsyncs right? [20:46:55] andrewbogott: personally, I'm okay with not migrating it. It's at best 4-5 hosts, and we can just add them manually later [20:47:17] Well, I either need to migrate existing data /or/ new installer code that creates the initial db [20:47:24] As it is, deploying to a new host just gets me 500s [20:48:13] andrewbogott: can I get you just an SQL statement + a commandline to execute that by hand? [20:48:16] that's trivial enough to do [20:48:20] sure. [20:48:22] moment [20:48:27] To create the initial empty db you mean? [20:48:39] ya [20:48:42] then it won't 500 [20:51:00]