[00:01:45] :(((((((((((((((((( [00:02:07] I point to a name though [00:06:00] I am wondering if I could setup iptables to have that public IP address to be rewritten to the internal IP address [00:07:13] you'd just be nating nat [00:07:50] like 208.80.153.219 (which is *beta.wmflabs.org pointing to the squid instance) to be rewritten to 10.4.0.17 which is the instance hosting squid [00:08:06] then I have to figure out how to puppetize that haha [00:15:13] Krenair: bleh [00:15:16] Krenair: fixes now [00:15:23] *fixed [00:15:37] seems filling up / so often likely corrupted the ldap server on nova-precise2 [00:15:52] it was saying an object didn't exist, even though it totally did [00:16:13] how did / get full? [00:16:13] those backups in /a or whatever it was? [00:16:17] yeah [00:16:19] I've fixed that [00:18:20] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - normal) [Bug 45868] let search indexer access the squid public ip - https://bugzilla.wikimedia.org/show_bug.cgi?id=45868 [00:21:33] hooray, my patch works [00:21:35] after a single fix [00:21:50] review, anyone? https://gerrit.wikimedia.org/r/#/c/52592/ [00:22:00] OpenStackManager fix to add admins to project pages [00:46:14] !log deployment-prep upgrading all instances [00:46:16] Logged the message, Master [00:56:52] hooray. admins are now shown on project pages [00:57:29] Ryan_Lane, does that dynamically update? [00:59:14] yep [00:59:27] the maintenance script was just needed to make it happen immediately [01:58:52] [bz] (NEW - created by: Krinkle, priority: High - enhancement) [Bug 34250] [beta project] Set up search (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=34250 [03:32:01] About 2,5 h ago, I received a watchlist mail for changed content on wikitech (http://pastebin.com/WUa20T8R) that claims the page was edited by 127.0.0.1 and the server name is "localhost". Is this a known issue? [03:33:37] yeah :P [03:41:12] legoktm: Do you have some bug number to watch? [03:41:19] lemme find it [03:41:42] i think its https://bugzilla.wikimedia.org/show_bug.cgi?id=43701 [03:47:50] legoktm: Hmmm. I don't think that's the same issue. The edits in question seem to really have been made from 127.0.0.1 (cf. pages' histories), so the more important issue seem to be the wrong links to http://localhost/. [03:50:15] File a new bug then? [03:50:47] I'm not too familiar with Echo :/ [03:51:35] As Ryan seems to be offline and andrewbogott_afk afk, that's what I'll do :-). [04:09:52] https://bugzilla.wikimedia.org/show_bug.cgi?id=45883 [04:33:19] Coren: around? i have a quick question about db replication [06:27:41] Anyone know why http://nagios.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=account-creation-assistance&style=detail is happening? [06:27:57] I've restarted NRPE on both instances [08:35:28] !log account-creation-assistance Going to try rebooting all systems to see if that fixes the NRPE problems. [08:35:30] Logged the message, Master [08:44:42] I can't apt-get upgrade instances in account-creation-assistance, brewster is refusing connections [08:47:14] !log account-creation-assistance Nope, reboots did not fix the errors. Interestingly, only application and database are showing errors (for load, disk, RAM, numprocs, and dpkg): "CHECK_NRPE: Error - Could not complete SSL handshake." Puppetmaster is reading all-green. [08:47:15] Logged the message, Master [09:08:36] wtf is going on [09:08:46] so many instances are down [09:08:53] !nagios [09:08:53] http://208.80.153.210/icinga http://icinga.wmflabs.org/ [09:10:45] FastLizard4 run sudo puppetd -tv [09:11:05] petan: Okay [09:11:15] petan: To fix which problems, though? :P [09:11:21] nagios [09:11:32] Okay [09:11:43] The server indicates that there was a successful puppet run 11 minutes ago, though [09:12:23] mhm in that case I have no idea what is wrong, but you should see it on your local logs [09:12:29] because that's problem of nrpe [09:12:42] maybe you changed some security rules? [09:12:48] there needs to be open port [09:12:52] so that nagios can see it [09:13:29] petan: I don't believe anything has changed in terms of security rules [09:13:34] However, I do see this in the puppet output: [09:13:34] Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/php5/php5-cli_5.3.10-1ubuntu3.5_amd64.deb Could not connect to brewster.wikimedia.org:8080 (208.80.152.171). - connect (111: Connection refused) [09:13:35] Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/php5/php5-mysql_5.3.10-1ubuntu3.5_amd64.deb Unable to connect to brewster.wikimedia.org:8080: [09:13:35] Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/php5/php5-curl_5.3.10-1ubuntu3.5_amd64.deb Unable to connect to brewster.wikimedia.org:8080: [09:13:35] Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/p/php5/libapache2-mod-php5filter_5.3.10-1ubuntu3.5_amd64.deb Unable to connect to brewster.wikimedia.org:8080: [09:13:35] Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/p/php5/php5-common_5.3.10-1ubuntu3.5_amd64.deb Unable to connect to brewster.wikimedia.org:8080: [09:13:35] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? at /etc/puppet/manifests/webserver.pp:79 [09:14:08] there is somethinh with network [09:14:39] petan: 5666 is the NRPE port, right? [09:14:45] yes [09:15:09] 5666 through 5666 proto tcp open to 10.4.0.0/21 [09:15:44] * FastLizard4 shrugs [09:19:45] ok, so which instance is it? [09:20:55] petrb@nagios-main:~$ ping account-creation-assistance [09:20:56] ping: unknown host account-creation-assistance [09:21:45] @labs-project-instances account-creation-assistance [09:21:45] Following instances are in this project: accounts-application, accounts-database, accounts-puppetmaster, [09:23:24] petan: accounts-application and accounts-database are the ones giving NRPE errors [09:23:30] Interestingly, accounts-puppetmaster is not [09:23:36] http://nagios.wmflabs.org/cgi-bin/icinga/status.cgi?hostgroup=account-creation-assistance&style=detail [09:24:10] try to restart nrpe then? [09:25:04] I rebooted the servers not too long ago, didn't fix the problem [09:25:11] But lemme try restarting nrpe directly [09:25:12] is nrpe running? [09:25:19] petan: You have more than one commit that you are about to submit, what can I do to remove this message? [09:25:34] matanya huh? [09:25:40] you is me or you? :P [09:26:04] I'm trying to push something, and I get this message [09:26:07] ah [09:26:12] I only want to push one thing [09:26:26] petan: Yes [09:26:30] not really a gerrit expert... [09:26:33] @seenrx demon [09:26:33] petan: Last time I saw ^demon they were quitting the network with reason: Remote host closed the connection at 3/8/2013 1:16:04 AM (08:10:29.6974470 ago) (multiple results were found: ^demon|lunch, ^demon|away, ^demon|busy, ^demon|dinner, ^demon|brb and 5 more results) [09:26:36] * matanya is a git noob [09:26:51] fastlizard4@accounts-application:~$ ps aux | grep nrpe [09:26:51] 4294967295 986 0.0 0.1 23260 1136 ? Ss 08:38 0:00 /usr/sbin/nrpe -c /etc/icinga/nrpe.cfg -d [09:26:53] matanya maybe gerrit doesn't accept more than 1 commit? [09:26:53] o_o [09:27:14] matanya try #mediawiki [09:27:14] I can override it, but I don't want to [09:27:20] thanks [09:27:26] matanya who knows if you really can :P [09:27:30] gerrit might reject it [09:27:42] :) [09:27:55] FastLizard4 interesting... [09:28:06] FastLizard4 I probably know what is problem then [09:28:12] petan: Yeah, on both -application and -database, the running UID is -1 [09:28:20] FastLizard4 problem is that you are running icinga version [09:28:26] which isn't maintained by puppet [09:28:30] nagios version is [09:28:48] Ahh, okay [09:28:50] puppet is changing /etc/nagios files [09:28:56] So how do I switch back? :P [09:28:59] petan: icinga was merged last night, wasn't it? [09:29:05] matanya I don't know [09:29:14] matanya but whoever merged it, they did it wrong I guess [09:29:48] FastLizard4 just copy all files from /etc/nagios/nrpe* to /etc/icinga [09:29:56] then restart it [09:30:14] petan: Okay, stand by [09:33:36] Aha [09:33:38] There we go [09:33:49] petan: I had to manually killall nrpe before starting the daemon [09:33:52] aha [09:34:01] The running one wasn't responding to my stop commands [09:35:25] Yup, fixed it on both [09:36:19] !log account-creation-assistance Fixed incinga reporting by `sudo cp -R /etc/nagios/* /etc/icinga/ && sudo killall nrpe && sudo /etc/init.d/nagios-nrpe-server start` on -application and -database. [09:36:20] Logged the message, Master [09:36:30] Thanks petan! :) [09:36:34] np [09:42:56] btw matanya if you find out how to do what you need pls add it: https://www.mediawiki.org/wiki/User:Petrb/Git_for_idiots [09:43:01] I <3 this guide [09:43:37] yeah, I found, thanks :) not sure what I did, LOL [09:44:44] from guide: "Pushing - This is a biggest evil of git for us, idiots :)" [09:44:53] lol [09:44:59] oh yes! I agree [09:45:19] and i'm sort of a master in svn, but git gives me a hard time [09:45:32] heh [09:45:55] well, I'll learn at the end, I guess [10:48:13] petan: how much space does the database server have? i just realized im importing a rather large amount of data and should check to make sure i dont hit the limit [10:48:24] database server == bsql01 [10:49:29] right now some 120gb I think but it can be increased [10:51:30] how do i check how much space a db/table is using? [10:52:29] right now i'm at 61k / 500k done with the import so... [10:54:29] oh [10:54:30] lol [10:54:40] i'm at like 59MB [10:54:44] i wont come even close [10:56:11] legoktm ssh to bots-bsql01? [10:56:17] you will see how much space left [10:56:26] if you want to know per db [10:56:47] SELECT table_schema "DB Name", sum( data_length + index_length ) / 1024 / 1024 "DB Size in MB" [10:56:48] FROM information_schema.TABLES GROUP BY table_schema ; [10:56:49] Usage of /: 66.2% of 9.83GB Users logged in: 1 [10:56:55] lol no [10:56:58] /db [10:57:00] not / [10:57:24] you mean [10:57:25] legoktm@bots-bsql01:~$ du -hs /db [10:57:25] du: cannot read directory `/db': Permission denied [10:57:26] ? [10:57:29] no [10:57:31] type df [10:57:48] or df | grep db [10:57:51] nothing called db [10:57:57] petrb@bots-bsql01:~$ df | grep db [10:57:59] oh wait [10:58:05] /dev/mapper/vg-lvol1 148897792 9739824 137201024 7% /db [10:58:07] yes [10:58:17] * legoktm trouts himself [10:58:31] btw [10:58:37] how so i see other users databases? [10:58:40] do they have to add me? [10:58:43] yes [10:58:55] they need to grant you access to db [10:59:01] so that you can see how much it uses [10:59:04] ok [10:59:05] using the query I sent you [10:59:11] but tbh I don't think it's secret [10:59:39] is the "_p" thing that the toolserver uses for public databases a ts thing or mysql? [10:59:39] | addbot | 4830.93750000 | [10:59:40] | information_schema | 0.14062500 | [10:59:41] | legoktm | 59.57812500 | [10:59:42] | mysql | 0.73980331 | [10:59:43] | performance_schema | 0.00000000 | [10:59:44] | salebot | 3614.26486206 | [10:59:45] | wiki | 6.16044617 | [11:00:09] I think it's ts thing [11:01:02] ts [11:01:53] can we get something like that here too? its much easier to name a database than fiddle with mysql settings [11:08:21] legoktm maybe yes... [11:08:30] should i file a bug? :P [11:08:38] if you don't want me to forget then yes [11:08:40] :D [11:10:01] yes, the toolserver gives read access to everyone to dbs whose name ends with _p [11:10:17] and each user has full access to tables which begin with u_ + its username [11:10:33] it's pretty easy [11:11:09] grant select, usage on _p%.* to '%'.'%' [11:11:19] but it's also pretty insecure :D [11:13:31] petan: how so? [11:13:48] legoktm if user who creates such a db doesn't know it... [11:13:54] [bz] (NEW - created by: Legoktm, priority: Unprioritized - normal) [Bug 45895] Use "_p" suffix on databases to make them readable by all users - https://bugzilla.wikimedia.org/show_bug.cgi?id=45895 [11:14:36] i doubt that will happen, just have a help page explain it like tswiki has [11:18:12] legoktm can you try to use _ptest [11:18:40] ok, but wont i be able to automatically see any db i create? [11:19:01] yes [11:19:11] but this one you should see as ewll [11:19:16] mysql> create database legoktm_ptest; [11:19:16] ERROR 1044 (42000): Access denied for user 'legoktm'@'%' to database 'legoktm_ptest' [11:19:19] ? [11:19:38] oh lol [11:19:41] I meant _ptest [11:19:45] not legoktm_ptest [11:19:51] oh lol [11:20:05] also when you try to create such a db, you need to prefix _ with \ [11:20:13] because _ is special symbol in sql [11:20:26] err dont think so [11:20:34] doesn't work? [11:20:36] ERROR 1044 (42000): Access denied for user 'legoktm'@'%' to database '_ptest' [11:20:37] mysql> create database \_ptest; [11:20:37] ERROR: [11:20:37] Unknown command '\_'. [11:20:37] ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '\_ptest' at line 1 [11:20:38] mysql> create database _ptest; [11:20:39] ERROR 1044 (42000): Access denied for user 'legoktm'@'%' to database '_ptest' [11:20:46] mhm [11:20:53] that database already exist - I created it [11:20:57] can you just try to use it [11:21:02] use _ptest; [11:21:07] mysql> use _ptest; [11:21:07] ERROR 1044 (42000): Access denied for user 'legoktm'@'%' to database '_ptest' [11:21:14] that's what I needed... [11:22:30] petan, that's a prefix, not a suffix [11:22:41] so you wanted a suffix? [11:22:47] I thought you want a prefix [11:22:50] read the bug summary :P [11:22:59] ah lol [11:23:11] i was wondering... [11:23:29] okay I now did it for _p [11:23:34] but I still think it will not work [11:23:58] legoktm try [11:24:00] use test_p; [11:24:12] nope [11:24:13] mysql> use test_p; [11:24:13] ERROR 1044 (42000): Access denied for user 'legoktm'@'%' to database 'test_p' [11:24:14] and undo the prefix grant? [11:24:28] huh? [11:25:15] I revoked it [11:25:29] ah, ok [11:25:33] GRANT SELECT ON `%\_p`.* TO '%'@'%' [11:25:41] this is what I see in show grants for '%'@'%'; [11:27:48] petan, on which server are you working? [11:28:08] bots-bsql01 [11:28:44] and why do i get this? [11:28:44] $ mysql -h bots-bsql01 [11:28:45] ERROR 1045 (28000): Access denied for user 'platonides'@'i-000000e5.pmtpa.wmflabs' (using password: NO) [11:31:22] petan, what if you do grant select, usage on `\_p%` . * to '%'@'%'; [11:31:26] argh [11:31:39] grant select on `%_p` . * to '%'@'%'; [11:33:01] I did select, usage [11:33:15] petan: Actually, looking at your patch on Gerrit, I think I found the real problem [11:33:26] I have the production IPs in my nrpe.cfg, not the labs IP [11:33:28] Platonides I executed exactly this [11:33:59] petan, I mean removing the usage [12:59:50] [bz] (NEW - created by: silke.meyer, priority: Unprioritized - normal) [Bug 45897] puppet - https://bugzilla.wikimedia.org/show_bug.cgi?id=45897 [13:58:49] petan: yes, true [13:59:00] um [13:59:06] ?? [13:59:07] :D [13:59:19] you are responding to which message [14:24:32] legoktm fixed [14:24:43] yay! [14:24:45] thanks :) [14:25:01] so just _p at the end? [14:25:06] yes [14:25:40] great [14:51:53] addrawr [14:51:59] . [14:52:03] @notify addrawr [14:52:03] This user is now online in #huggle so I will let you know when they show some activity (talk etc) [14:53:45] he should be back in an hour [14:59:55] btw Platonides I like you found a security hole in mysql and far bigger security hole you overlooked [15:00:21] which was to sudo su on any application server where everyone has root to get their credentials [15:00:35] I managed to fix that before anyone notice [15:00:36] :D [15:10:15] petan: im home ;p [15:10:32] I am wondering if we restart or not restart the sqld lol :/ [15:10:39] noooo [15:10:45] :P im using it too much now ;p [15:10:46] how it affect ur bot? [15:10:53] because that is only service running there atm :D [15:10:54] errrr [15:10:57] heh [15:11:01] legoktm no worries [15:11:02] is that bsql01? [15:11:09] I am just thinking of it in this moment :D [15:11:12] not doing it [15:11:25] addrawr I am still trying to figure out how that memory thing works [15:11:31] xD [15:11:31] virtual memory is 17GB now [15:11:36] i have an active db import running thats going to be done in a few more days hopefully [15:11:39] but physical is only about 6gb [15:11:47] legoktm aha it's ok then [15:11:53] don't worry [15:11:55] :) [15:11:57] we can restart it later [15:12:23] wow, it would appear I am skating near the edge of memory land in bnr1 [15:12:23] http://ganglia.wmflabs.org/latest/?r=2hr&cs=&ce=&c=bots&h=bots-bnr1&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [15:12:41] a bunch of that is probably mine [15:12:52] lol nice load [15:13:13] we could actually create bnr2 but I would like to setup some scheduling before we do that [15:15:16] would be nice if we eventually did get some kind of job queue thing, i have like 10 screens running right now [15:15:24] yes I know [15:15:30] we could just setup that grid thing [15:15:31] :P [15:15:54] petan: legoktm I have a 380 line cron ;p [15:16:01] >.< [15:16:03] only 380? :P [15:16:12] xD [15:16:23] well 100 are commented out from my enwiki bot thats not currently running [15:16:24] but pah! [15:16:46] bbl [15:22:48] [bz] (NEW - created by: silke.meyer, priority: Unprioritized - normal) [Bug 45897] puppet has a problem with versions - https://bugzilla.wikimedia.org/show_bug.cgi?id=45897 [15:34:16] !log bots create new instance to play with grid scheduler [15:34:18] Logged the message, Master [15:38:50] :D [15:43:49] addrawr can I delete all your db's from sql3? [15:44:02] yep :) [15:44:05] ok [15:54:18] addrawr I will also build a kernel 3.8.2 for bsql01 [15:54:32] for performance reasons [15:54:39] btrfs had tons of improvements [15:54:44] and version 3.2 is very old [15:54:54] even some critical bugs [16:02:22] kk :) [16:45:09] @seen Beetstra [16:45:09] petan: Last time I saw Beetstra they were quitting the network with reason: Quit: Leaving N/A at 3/6/2013 1:38:21 PM (2.03:06:48.6771700 ago) [16:47:06] @notify Beetstra [16:47:06] I will notify you, when I see Beetstra around here [17:08:50] !log wikidata-dev wikidata-testrepo: Due to a version problem puppet sometimes refuses to run because it doesn't downgrade. To solve this, remove and then reinstall php5-mysql manually. [17:08:52] Logged the message, Master [17:19:34] I am not able to ssh -p 29418 to gerrit, and I have been troubleshooting this for awhile now. Could someone look at the ssh log on the server to see what error messages are appearing when I ssh in ? [17:21:30] I have a lab account setup, rachel99, and my public key is in there. Is there someone specific I should ask this question to? To check the ssh log on the server? [17:21:52] A wild Coren appears! [17:22:10] Hello- can you help me with this? [17:23:01] I am working with chrismcmahon on the QA testing, if that helps at all. [17:23:03] I don't think I can look at those logs, but I can try to help troubleshoot. [17:23:41] hi rachel99, thanks for asking here [17:24:20] Have you tried to ssh with -v so we have a better idea of where it fails? [17:24:54] Coren: yes I have done that with -v -v -v? Do you want me to dpaste the output? [17:25:24] rachel99: Just the last half-dozen lines should suffice, that should be enough to see the last thing it tried before giving up. [17:26:34] http://dpaste.com/1016287/ [17:28:13] Hm. [17:28:59] Anything jump out at you? [17:29:32] Well, it doesn't recognize your key clearly. You're saying that this is the key you use to log on bastion? [17:29:59] You did login to gerrit and set your key there as well, yes? [17:30:07] (Through the web interface) [17:30:25] Yes, I set it on gerrit. Haven't set it on bastion, I don't think. [17:31:04] Do you have the same username on gerrit as you do on labs? [17:31:16] Yes, its rachel99 [17:32:50] Well, you're in the bastion project. It'd probably help if you tried to log on there with ssh to see if your key is accepted. [17:33:41] is https://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use still 'draft'? [17:33:47] Coren: ^ [17:33:50] (No Ryan_Lane) [17:34:10] YuviPanda: Yes, I know Luis is currently looking at it and making something nicer. [17:34:24] ah, okay. [17:34:25] Coren: So, I would add it to the labs machine, and then ssh to bastion? Any particular port? [17:34:52] Just the default. You can add your key on wikitech. [17:35:03] [[Special:NovaKey]] [17:35:33] * Coren tries to hit git himself. [17:36:42] Yeah, okay, there's probably no problem with gerrit itself, my own key gets me through. [17:36:55] Coren: Ok. Will try it. I will need to run in a few minutes, so I will try it, and get back to you later this afternoon. (if you are here) [17:37:17] rachel99: I'm going to be here until at least 00:00 UTC. Don't hesitate to look me up. [17:37:28] Ok, Thanks. [17:39:23] Coren: what's your TZ? [17:50:48] YuviPanda: Nominally EST [17:51:00] YuviPanda: (UTC-5) [18:26:17] [bz] (NEW - created by: Antoine "hashar" Musso, priority: Unprioritized - major) [Bug 45908] review lsearch-global.conf for beta context - https://bugzilla.wikimedia.org/show_bug.cgi?id=45908 [18:29:28] thanks Coren ^^ [18:29:49] ... for not having been able to help yet? [18:29:52] :-) [18:31:31] Coren: just for being nice. I've been working with her, but I tried everything I could think of. ^demon checked and she seems to be OK on the gerrit server side. [18:32:45] Yeah, that side seemed to be okay. With the test on bastion we'll have a better idea if there is something wrong with her public key. [18:35:27] [bz] (NEW - created by: Krinkle, priority: High - enhancement) [Bug 34250] [beta project] Set up search (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=34250 [18:35:53] Coren: I know she's on Windows, which might be a factor. I haven't done ssh in anger on a Win box in a decade. [18:46:31] ... it was clearly openssh logs she gave me. Cygwin? [18:46:50] * Coren would recommend putty to Windows users as a rule. [18:47:02] Oh, wait, she's using it to pull git. Nevermind. [19:13:33] Ryan_Lane, if you do get spam with autoconfirmed protection you could always add the shell group as a protection level and use that [19:19:58] Krenair: is that possible? [19:20:14] I didn't see a way to allow groups other than admin and autoconfirmed in the protection interface [19:20:56] ptwiki does it with their 'autoreviewer' group [19:21:09] See https://www.mediawiki.org/wiki/Manual:$wgRestrictionLevels [19:21:28] petan: Did you really send me a message on 'Badoo' or is this spam? [19:21:38] spam [19:21:45] Damianz ^ [19:21:54] * Damianz terminates [19:22:04] that freaking shit hacked to my gmail using oauth and spammed all my contacts [19:22:49] rofl [19:25:51] bugger, this video is in french :( oh well, food then I'll move cb [19:27:45] yeah, I got one from him too :) [19:27:58] petan: have you not remove the authorization yet? [19:28:01] *remvoed [19:28:03] fuck it [19:51:54] https://pbs.twimg.com/media/BE0Id46CUAIP_x7.jpg:large < interesting claim [19:55:06]