[00:09:40] Betacommand tyvm [00:10:09] * Betacommand gumbles about ToAruShiroiNeko not writing that down last time [00:11:15] Sorry, I was gullable enough to think it would be fixed instantly :( [00:21:35] Coren: I should probably know this already, but how can I scp files between pmtpa and eqiad? I tried pushing from pmtpa to eqiad but got an "unknown host" error for the eqiad address. Same thing from bastion in pmtpa. bastion-eqiad.wmflabs.org gives "unknown host" for the pmtpa side. [00:24:37] Coren: Never mind. I figured it out. I can pull to the bastion and push from there just not an end to end copy. [00:34:24] bd808: You can also do end-to-end, but DNS won't work for the 'fake' .wmflabs addresses so you have to use IP [00:48:27] Someone already having experience with self-hosted puppetmasters in eqiad? I get a lot of "Error 400 on SERVER: Not authorized to call find on /file_metadata/files/ssl/wmf-ca.pem" and ... labs-puppet-key with that, but too tired to debug this today. [04:59:09] does anyone else have problems with the webproxy for instances on eqiad? [05:00:20] I got bad gateway errors, deleted the proxy entry and recreated it, then it worked for a while and now it is back to throwing 502s again [05:29:57] Cyberpower678: Here they are! [05:30:12] andrewbogott, yes. More than 20. :p [05:30:13] Creepy, right? [05:30:22] That's really wierd. [05:30:46] morebots, are you here too? [05:30:52] Coren, so I get mysqli_connect(): (HY000/2003): Can't connect to MySQL server on 'tools-db' (99) [05:30:55] nope :( [06:32:07] hi, i just migrated my tool to eqiad. I can't find there the source for pywikpedia in the shared directory (my bot depend on pywikipedia). Should I clone it locally? [06:45:48] according to ...MIGRATE STATUS pywikipedia already moved, but probably it isn't accessible (ls doesn't show files but only directories) [06:56:41] (solved: cloned it locally. using shared isn't currently possible) [07:19:54] My mwui instance (a simple single node wiki proxied to http://mwui.wmflabs.org/wiki/Main_Page) stopped working again. [07:20:02] It's on eqiad, and I'm getting 502 Bad Gateway. [07:20:09] I used to be able to ssh to it, but now I can't. [07:23:56] Is bastion-eqiad.wmflabs.org the right bastion for me to use? [07:40:29] superm401: that's the right bastion, sure. [07:40:39] superm401: but, the 502 is… well, let me look. [07:40:59] superm401: what is the project name? [07:41:16] editor-engagement [07:41:56] OK, so -- we're talking about two different problems, right? ssh and also http? [07:42:31] The ssh issue is most likely this one: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Howto#Security_Groups [07:42:36] I can fix that for you if you would like. [07:43:10] superm401: ? [07:43:39] andrewbogott, yeah, both, and both were working before. :) [07:43:59] When you say 'working before'... [07:44:07] you mean with the new eqiad instance? [07:44:12] andrewbogott, yes. [07:44:24] But not with bastion-eqiad. [07:44:40] I think it was bastion2.wmflabs.org [07:44:51] Ah, sure, that's the difference. For that bit at least. [07:45:01] Do you want to read and understand the above link, or just have me fix it? [07:45:07] (This is ssh. The http thing I don't know about yet.) [07:46:06] andrewbogott, which group name do I add it to? [07:46:29] you'll want to ssh to everything, so change the rule in 'default' to allow from 10.0.0.0/8 [07:49:54] andrewbogott, done, how long is it supposed to take? [07:50:04] for ssh? Should work right away. [07:52:10] andrewbogott, got it. My local ssh config also wasn't quite right. [07:52:36] cool. OK, I'm working on the proxy right now anyway so I'll look at your http issue as well... [07:52:50] It says 'System restart required', FWIW. [07:53:16] andrewbogott, I can reboot through the wikitech UI if that doesn't interfere with you. [07:53:35] hang on just a second... [07:54:32] superm401: there's a dns issue… I think you've done everything properly with the proxy. [07:54:34] I'm not sure what's up [07:54:41] If you need to reboot for other reasons then that's fine. [07:55:27] superm401: fixed. Although I fear that DNS is dieing off every day or so :( [07:55:31] Anyway, working for you now too? [07:56:01] andrewbogott, yep, thanks for your help. [07:56:11] superm401: do you mind helping me test something? It should be easy... [07:56:18] andrewbogott, sure. [07:56:22] I've just changed wikitech to allow you to use a proxy in eqiad for eqiad instances... [07:56:31] So, want to delete the current proxy for that box and create a new one in eqiad? [07:56:35] Should be pretty obvious how to do it. [07:56:40] Okay, I better not tell anyone it's fixed yet. ;) [07:57:03] Yeah, moving to a different proxy will require dns to propagate again... [07:57:21] I didn't even realize it was proxying between data centers before. [07:57:21] (It can have the same name of course.) [07:57:36] I'll do a fun test first... [07:57:37] Until an hour or so ago there was only one proxy, in tampa. [07:57:40] Now there are two. [07:58:24] And now there are two proxies, both with mwui.wmflabs.org. [07:58:30] Probably shouldn't allow that, right? [08:00:14] ^ andrewbogott [08:00:41] superm401: Huh, it should already prevent that. I'll have a look. [08:05:38] superm401: working ok other than that? [08:06:05] andrewbogott, yep. [08:06:39] cool, thanks for testing! [08:06:47] I'm going to disable creation of pmtpa proxies in just a minute... [08:07:00] No problem, looks like eqiad won, BTW. [08:07:12] I bet it's random :( [08:16:17] isn't round-robin DNS *fun*? [10:29:11] hoi, the backups that exist in /public/datasets/public/wikidatawiki are not present on eqiad [10:29:30] they are run by a user called "backup" [10:29:53] can someone fix this so that the new environment works fine ? [11:54:54] I didn't follow updates to Labs / Tool-Lab recently, but what's *OLD* in bash prompt? [11:54:56] local-liangent-py@*OLD*tools-login:~/scripts/updatedyk$ [12:03:22] and regarding migration: is it fine (and actually better) not to touch them if I want my instances archived [12:04:02] so they don't need to keep running but I'll be able to get old data back at some point in the future [13:43:52] andrewbogott_afk: The nova-network dhcp/dns thing seems to be full of fail. [13:44:12] dhproxy. [14:07:37] Hi, I tried to migrate some tools (migrate-tool), but the webservice doesn't seem to restart. Do I need to do something else than "finish-migration"? [14:08:12] Darkdadaah: Well, possibly 'webservice start' [14:08:15] :-) [14:08:33] Ah, I did try that too. [14:08:45] Oh? That /should/ have worked. Let's see. [14:09:12] Oy! [14:09:21] We're already out of slots on webgrid. :-) [14:09:33] (Not long or hard to fix) [14:09:44] Ouf [14:17:34] Darkdadaah: Should be started now. [14:19:51] Ok, it seems to work :) [14:23:19] Coren, what's dhcp/dns doing? [14:23:28] andrewbogott: Sucking. [14:23:35] Could you be more specific? [14:24:03] andrewbogott: We had the same problem in pmtpa, remember? Where the dhrelay thing just drops a fraction of requests to /dev/null when it just vaguely looks like it's being lightly loaded. [14:24:41] I only barely remember this... [14:27:08] …which makes me think I must not have been the one who fixed it. [14:27:16] Coren, Do you remember what the solution was, then? [14:27:21] ok andrewbogott, trying proxy again [14:27:27] getting 404 from nginx right now [14:27:40] inside network direct to instance works [14:27:49] curl http://archiva0.eqiad.wmflabs:8080 [14:28:01] but archiva.wmflabs.org does now [14:28:02] not [14:28:03] * [14:28:21] ottomata, what project? [14:28:21] andrewbogott: There wasn't; or at least I never found one. All I did was alleviate the problem by reducing the number of queries by setting up a local DNS in some projects, and adding entries in /etc/hosts in tools. I was /hoping/ that the new version woulld fix this. [14:28:34] analytics [14:30:06] ottomata, archiva.wmflabs.org looks to be working for me. Or, at least, I get 'Apache Archiva' and an icon up top. [14:30:40] Coren: ok… so which components are involved in this? It's not just pdns? [14:31:00] (Sorry to be clueless...) [14:31:46] hm [14:31:49] not working for me [14:31:53] i get 404 [14:31:54] from nginx [14:31:58] Coren: how do I know what is the new name of my database after migration? [14:31:59] andrewbogott: I don't think it's pdns at all; I think it's dnsmasq [14:32:02] is it hitting the wrong IP, maybe my dns is cached? [14:32:23] ottomata: probably you're hitting the old proxy. Or I am. [14:32:43] hmm ok i think it might be my dns, i just curled from another server and it worked, k [14:33:02] Darkdadaah: The prefix will be the same as the username in replica.my.cnf [14:33:18] Darkdadaah: sXXXXX [14:33:40] ottomata: if you get a 208.80.153 IP, that's still in tampa. [14:33:44] .155 is eqiad [14:33:45] Coren: Yes but, what about the suffix? I used to have "localanagrimes" as a name. [14:34:24] yeah i got that locally [14:34:25] k [14:34:29] trying to flush os x cache... [14:34:41] thanks andrewbogott, looks like it is working after all, on it.. [14:34:53] Coren, so… is the dns problem causing immediate failures, or is it a longer-term/capacity issue? [14:35:06] (Because I have an immediate problem which I want to change the subject to :) ) [14:35:24] andrewbogott: I think it's causing infrequent issues at this time; but it'll just increase [14:35:39] Ok, I've certainly seen infrequent dns failure as well :( [14:35:51] same here [14:36:06] But I will have to study up in order to understand what component to blame. [14:36:12] Darkdadaah: I see 's51045__anagrimes2' [14:37:00] Coren: this is an experimental and empty one that I just created. [14:37:56] (and I can access this one in eqiad) [14:38:19] Darkdadaah: That's the only one the automated system found. You'll have to dump and restore any others you might have had that had different naming. [14:38:51] Ok. I suppose it's because it was created before we used the username for the db name. [14:39:12] Darkdadaah: Probably. You'll also have to edit the dump to fix the name of the database, but that's easy. :-) [14:39:49] andrewbogott: So, what subject did you want to change to? [14:40:20] Coren: I emailed. Migrated images don't have /dev/sdb after I copy them over [14:40:31] andrewbogott: Ah that. [14:40:34] I'm thinking since you've done some recent thinking in that area… [14:40:35] Ok, in the meantime I move back the old public_html to use the working db. [14:41:36] andrewbogott: I didn't think so much as observe that eqiad sets up "large vda" rather than "small vda and vdb for the rest". It's easy to adapt, then. [14:42:01] andrewbogott: But if copying the image over loses the vdb, that's a more complicated matter. [14:42:44] andrewbogott: Do you have a running migrated instance that losts its vdb I can look at? [14:42:47] sure. [14:43:03] wikisource-dev.eqiad.wmflabs [14:45:19] hiya YuviPanda? [14:47:43] andrewbogott: And things were going so wel... as far as I can tell, the method you use to copy images doesn't actually carry the /dev/vdb with it at all. [14:48:45] andrewbogott: All the image seems to have is the small (16G) vda [14:59:52] Coren, sorry, network downage [15:00:22] So… the files that I copied over are… disk.local, 470M and disk, 8.6G [15:00:30] I would expect that that latter is /dev/vdb [15:00:39] Does that seem likely or unlikely? [15:01:13] ... that seems smallish. [15:01:29] 470M in particular is too small to be vda [15:01:34] hm, ok. [15:04:02] hi ottomata. [15:04:04] ottomata: sorry, only spoardoically available at the moment :( [15:04:39] that's ok, i was just wondering about the ssl proxy you set up that we are usign for wikimetrics, i thought for a second it wasn't working in eqiad labs, but I think it is [15:04:45] so , never mind! :) [15:04:55] ottomata: ah :) [15:05:03] ottomata: ok! :) [15:05:03] ottomata: good to know! [15:06:44] well, wait -- coren, it's copy-on-write. So the 470M should just be the diff. [15:07:31] andrewbogott: Hm... good point. [15:07:47] andrewbogott: Yeah, taking cow into account, the sizes would make sense. [15:08:23] I see another instance where 'disk' is 9g and 'disk.local' is 20G [15:08:59] So... [15:09:20] Probably the same reason why new instances didn't have /deb/vdb. Whatever that is. [15:32:35] are the dumps ('datasets') already in eqiad? [15:37:00] gifti: no, not yet [15:37:12] is there an estimate? [15:37:17] i heard it's 95% done [15:37:26] within March i'd say [15:37:30] uh, ok [15:37:34] who's responsible? [15:37:37] apergos [15:37:49] thx [15:37:53] yw [15:38:16] :) [15:44:47] andrewbogott: I don't think it's the /same/ reason; instances in eqiad get created with a variable-sized vda so it /is/ taking the instance size into request, just not the same way as in folsom [15:57:52] andrewbogott_afk: More importantly, we need to figure out a way around/through this. Clearly, creating the instance in eqiad first then overwriting the image has openstack "fix" the image to fit the disks. [15:58:05] (As they were created) [15:58:20] * Coren ponders. [15:58:54] Obviously, people who create new instances won't have issues; they can simply use the labs_lvm thing and get their storage. [15:59:40] (Which means that making it the default might actually make sense then) [16:06:43] gifti: https://bugzilla.wikimedia.org/62296 [16:06:46] ottomata: Have you tried running a self-hosted puppetmaster in eqiad? It fails for me with "Error 400 on SERVER: Not authorized to call find on /file_metadata/files/ssl/wmf-ca.pem" & Co. Any ideas? I set the IP subnet for eqiad in master.pp, but that didn't help. [16:07:00] How many instances are we talking about, and what constitutes an instance besides a tar ball of the file system? [16:08:50] hm, i have tried it and ahve not seen that [16:09:01] Coren, I'm stuck. [16:09:47] Coren, I seem to be getting mysqli_connect(): (HY000/2003): Can't connect to MySQL server on 'tools-db' (99) at random since the move to eqiad. [16:10:13] ottomata: One moment, I'll pastebin the log. [16:11:02] scfc_de: are you doing self hosted puppet with no special puppetmaster setting? [16:11:08] i.e. locally hosted self hosted puppet? [16:11:21] (i haven't tried it in eqiad with a remote self hosted puppetmaster yet) [16:11:45] Coren, any ideas? [16:12:31] Oh, sorry, should have been clearer: role::puppet::self with $puppetmaster = FQDN (local for the moment, but planned as a puppetmaster for other instances later). [16:12:41] Let me see if $puppetmaster = empty works. [16:13:09] ah ok, yeah i haven't tried setting that yet [16:13:21] * Coren reads scrollback. [16:14:03] Cyberpower678: "random"? [16:14:06] ottomata: Okay, clearing $puppetmaster on an instance with problems doesn't seem to be a good idea, let me set up puppetmaster3 :-). [16:14:21] Cyberpower678: Like, it works, then fails, then works again? [16:14:22] Coren, it sometimes works and sometimes doesn't [16:15:00] Coren, It doesn't work at all in a different but similar script. [16:15:17] $dblocal = mysqli_connect( 'tools-db', $toolserver_username, $toolserver_password, 's51059__cyberbot' ); [16:15:17] $dbwiki = mysqli_connect( 'enwiki.labsdb', $toolserver_username, $toolserver_password, 'enwiki_p' ); [16:15:27] $dblocal fails [16:15:27] scfc_de: worked for me yesterday or so, i put "localhost" in the form [16:15:32] $dbwiki works [16:15:33] and puppet::self worked [16:15:50] ha, yeah scfc_de, i wouldn't recommend that :) [16:16:02] you could get it to work eventually (clearing the setting), bu tit would be a pain [16:16:22] yeah so, mutante 'localhost' and empty are equivalent [16:16:27] it will infer localhost if you leave it empty [16:16:35] gotcha, ok [16:16:40] Coren, ^^^^^^^ [16:16:46] putting FQDN of a node there allows you to configure other labs instances as puppet clients of a self hosted puppet master [16:16:47] Cyberpower678: Yes, yes, I've read. [16:16:52] Ok. [16:17:00] Cyberpower678: Does dblocal fail always, or intermitently? [16:17:04] if the value there matches $::fqdn when puppet runs [16:17:09] then puppet will set up the node as the puppetmaster [16:17:10] Coren, it seems to be intermittent. [16:17:36] DNS related? [16:17:44] * Coren wishes the error message was less stupid than just 'can't connect' [16:18:05] couldn't the intermittent DNS issue also just cause this, can't resolve = can't connect? [16:18:18] mutante: It could; I wish I could know. [16:18:22] Try it with tools-doesnotexist? [16:19:00] Cyberpower678: if you type "host enwiki.labsdb" on that shell and repeat it a couple times, does it always return an IP? [16:19:23] mutante: enwiki.labsdb is resolved in /etc/hosts. [16:19:45] eh, then "tools-db" [16:19:49] So is tools-db actually. [16:19:49] the one that fails [16:19:53] So no dns. [16:20:03] ok [16:20:11] (tools-db is an alias for tools.labsdb) [16:20:14] unless nsswitch it set to ask DNS before files [16:20:49] Coren, http://tools.wmflabs.org/cyberbot/botlogs.php?bot=CyberbotII&task=spambot&type=err shows the last 10 MB of the stderr logs generated when in operation. Scroll almost to the bottom. You'll see it see all of the attempts it made and other time the bot completed a successful run. [16:20:57] mutante: Good idea. But "hosts: files dns" [16:21:07] hmm,ok [16:21:45] "Can't connect to MySQL server". What a useful, Microsoft-like error message. [16:22:05] "An error has occured." [16:22:15] :p [16:22:32] Can you reproduce the problem by hand at all? [16:22:41] does the GRANT for that contain a from IP range, and that is just Tampa? [16:22:43] mutante, Coren typing "host tools-db" returns nothing [16:23:07] Cyberpower678: host only checks DNS. [16:23:41] Coren, I just did what mutante asked me to do. [16:23:58] That's because he didn't know that those came from hosts. [16:24:01] Cyberpower678: it was a way to test DNS before i knew you had it in /etc/hosts [16:24:14] that's when i thought those problems could be related [16:24:25] show global status for tools-db says [16:24:27] | Aborted_clients | 505 | [16:24:28] | Aborted_connects | 184 [16:24:28] mutante: My automatically created grants all have @'%'. So not that either. [16:24:36] A while ago we had some connection issues with the replicas, and the error was in the DB servers being overloaded or something like that. Don't remember the details, but if tools-db is now the same setup, Sean might be of help. [16:25:12] scfc_de: Hitting the connection limit is usually simple to debug because you normally get a specific message for this; but I'll check. [16:25:50] It was something different (network?). IIRC January or so. [16:26:03] Coren, one of my scripts fails to connect to it entirely. I tried copying and pasting the mysqli_connect command from the working script to the broken script, but it's still not working. [16:26:33] so, there is a documentation for intuition on tools/help. how can i use translatewiki without intuition/php? [16:27:16] i can't detect the underlying mechanism [16:27:18] Coren: is there documentation somewhere about how to set up an app to use our ldap? [16:27:47] Cyberpower678: for it works fine with hte test script https://tools.wmflabs.org/tools-info/?dblist=tools-db [16:28:00] ottomata: I don't know; I don't think so (at least I've never seen it). [16:28:29] he could likely copy from Icinga login [16:28:44] ottomata: [16:28:49] Cyberpower678: repeatedly called w/o connection errors [16:28:55] at least some other config to copy from i suppose [16:29:09] uses same LDAP for auth [16:29:21] Cyberpower678: I've been trying to get the error repeatedly as well, and I'm got getting issues. [16:29:22] hedonil, then perhaps somebody could look at my replica.my.cnf file? [16:29:31] if it uses Apache that is [16:29:42] mutante: ? [16:29:48] oh [16:29:49] LDAP [16:29:50] yeah [16:29:55] ottomata: you want to setup an app to use LDAP for auth, right [16:30:00] It makes no sense for one script to work and another to fail. [16:30:04] yes, trying to see if I can make my archiva setup in labs use it [16:30:06] and it made me think of Icinga setup in puppet [16:30:07] Who knows how many databases are copied at various moments by the migrate-tool script; so there may be peaks. [16:30:14] so that should be the Apache module [16:30:16] for LDAP auth [16:30:23] Cyberpower678: Well, unless you have some bug in the way you read your credentials from the replica.my.cnf? [16:30:27] i'm a little worried about the proxypass [16:30:29] mod_auth_ldap i thik [16:30:31] since that comes from private [16:30:36] not sure if I need that [16:30:43] scfc_de: It's one-at-a-time on purpose to avoid exactly that. [16:30:54] Coren, I haven't changed it. It's always been the same so far. [16:30:55] Cyberpower678: I use the brand new replica.my.cnf one [16:31:02] hedonil, same [16:31:25] database.inc reads the replica.my.cnf file. [16:32:14] does it work manually without reading any credentials from files? [16:32:14] Cyberpower678: cwd issues? Does it read the file with an absolute path? [16:32:54] mutante: i dunno, i'm not entirely sure what port to connect to [16:32:57] or server [16:33:02] virt1000.wikmedia.org [16:33:06] port 1382? [16:33:09] 1389* [16:33:09] ? [16:33:13] Coren, no [16:33:30] $toolserver_mycnf = parse_ini_file($_SERVER['HOME']."/replica.my.cnf"); [16:33:30] $toolserver_username = $toolserver_mycnf['user']; [16:33:30] $toolserver_password = $toolserver_mycnf['password']; [16:33:30] unset($toolserver_mycnf); [16:35:55] If it was the same issue as some weeks before (db connection aborted due to network issues) we'd have seen another error message (...lost connection during etc) [16:37:29] * Coren tries to reproduce the error. [16:37:35] ottomata: actually i meant Cyberpower678 for that one:) but the LDAP port should be 1389 [16:37:40] opendj is configured to listen on ports 4444, 8989, 1636, 1389. [16:38:54] Cyberpower678: how about mysql -u user -p -h toolsdb enwiki or so [16:39:04] and then interactively entering pass [16:39:17] Cyberpower678: That's not the bug but you should use process_section in your parse_ini_file as there could be more than just the [client] section. [16:39:37] mutante, what good will that do? [16:39:48] * Coren tries to go see if the logs on the DB are any more informative. [16:40:21] Cyberpower678: ruling out possible issues with include the credentials from files etc [16:40:28] just to get closer to the root cause [16:41:43] SQL wokrbench seems to work with those credentials. [16:41:52] hm, mutante, am I allowed to talk to ldap from labs? [16:42:11] Cyberpower678: What's your tool's username? [16:42:22] cyberbot [16:42:29] on eqiad [16:42:34] ... DB username. :-) [16:43:38] Oh, seriously? That is SO useful mysql! [16:43:38] Coren, oh. Be more specific. :p S51059 [16:43:39] ottomata: i think you should be, but maybe not from eqiad instances yet? security group rules or so? [16:43:42] Mar 7 12:57:20 labsdb1005 mysqld: 140307 12:57:20 [Warning] Aborted connection 275787 to db: 's51059__cyberbot' user: 's51059' host: '10.68.16.39' (Unknown error) [16:44:04] "(Unknown error)" [16:44:14] yea, -u [16:44:15] :p [16:44:16] * Coren headdesks. [16:44:51] likes the "unknown" part [16:45:23] * Cyberpower678 wonders how an error could be unknown, but still be recognized as an error. [16:45:41] hm, mutante, i think security groups control incoming connections [16:45:43] not outgoing, [16:45:45] right? [16:45:46] well it knows it's an error it just doesn't know which kind:) [16:45:51] and you are rightr, I can connect to 1389 from a pmtpa instance [16:46:08] ottomata: but something must come back from LDAP to you incoming as well [16:46:24] estabilshed shoudl be allowed, right? [16:46:32] its not like the ldap server is going to open new sockets to my instance [16:46:33] ? [16:46:46] also, there are no 1389 security policies in pmtpa (in my project) [16:46:49] or in eqiad [16:46:51] unless it's set to allow all the related traffic, like ESTABLISHED,RELATED -j ACCEPT [16:46:53] but it works from pmtpa [16:47:33] well, then, i dunno, something networking related with eqiad [16:48:11] tcpdump? [16:49:19] ottomata: could the LDAP server itself be firewalled to only allow from pmtpa? [16:49:30] which you would not see in security groups in webui [16:49:49] UHHH, now it is letting me telnet [16:49:52] it was nto before [16:50:59] Cyberpower678: Google shows that there are some buggy php extensions that can interfere with database streams in some cases. Are you using geophp in those scripts that fail? [16:51:23] Coren, not likely. I'm using mysqli [16:51:58] ottomata: odd, we keep having intermittent issues it seems [16:52:07] What's mysqli? [16:52:15] yeah, i've noticed my proxies someimtes work and sometimes not [16:52:18] Coren, my script should've just flooded some connections into the log. [16:52:23] Reconnecting to local DB... [16:52:23] Reconnecting to local DB... [16:52:23] Reconnecting to local DB... [16:52:23] Reconnecting to local DB... [16:53:02] mysqli replaced mysql_connect etc [16:53:14] might be worth seeing what happens with the old method..shrug [16:53:35] but when you use that PHP will tell you it's deprecated and you should switch ,,juts warnings though, still WFM [16:54:09] !newtoollabs [16:54:11] Logs actually show three users getting those errors. [16:54:19] !newlabs [16:54:19] The tools project in labs another version of toolserver. Because people wanted it just like toolserver, an effort was made to create an almost identical environment. Now users can enjoy replication, similar commands, and bear the burden of instabilities, just like Toolserver. [16:54:21] But only those users. [16:54:57] Coren: So, what do they have in common? [16:54:58] Switching to my phone [16:55:36] hedonil: Absolutely nothing afaict. [16:55:44] I'm an iPhone. :p [16:56:56] CP678|iPhone: admit it, you compiled your own mysqli driver and it doesn't work [16:57:01] Please ping me for my attention. [16:59:23] hedonil: if I knew how to do that, I would've made work over SSH connections so I could debug DB dependent scripts on my computer more easily. [16:59:57] CP678|iPhone: hehe [17:00:39] I have no idea what this could possibly be. The client whines that it doesn't work without details, and the DB simply says that something went wrong without detail. [17:00:44] ottomata: role::puppet::self with empty $puppetmaster works, you're right. I'll file a bug for the multi-instance bug. [17:01:05] ottomata: Re LDAP, if it is the same server as queried by "ldaplist -l passwd", this seems to work from eqiad flawlessly. [17:02:22] CP678|iPhone: Please try from the command line with the mysql client. [17:02:28] Coren: makes it a little hard to fix doesn't it. [17:02:44] Coren: no access right now. [17:02:54] CP678|iPhone: so do that when you're back home? [17:03:11] valhallasw`cloud: no time. :/ [17:03:32] or, you know, tomorrow, or next week, or whenever you have time. [17:05:42] Coren: CP678|iPhone: here is an interesting thread http://lists.mysql.com/mysql/204830 [17:06:35] yeah scfc_de, it is working right now, now trying to figure out how to configure my app [17:06:36] ldap [17:06:37] tells about 'running out of available outgoing ports' [17:08:11] Coren: CP678|iPhone: maybe not properly closed connections or use of persistent-connections (although I can't see the latter in the cb-script) [17:08:37]