[00:47:58] hello anyone there to help us solve issues on labs? [00:48:26] we still cannot connect to dbs and it is likely due to some ip tables configuration we do not have. [01:02:08] you can always copy the labs iptables config [01:02:35] but i think it was the hosts file that changed [01:04:12] *tool labs iptables config [01:26:36] gifti: and any idea where i could get that one? [04:16:01] Is it possible to run a .sh script via CGI on Tools (via lighttpd)? [04:18:03] The script "http://tools.wmflabs.org/ext-lnk-discover/cgi-bin/server.fcgi" returns "403 Forbidden" (it is a #!/bin/sh and prints "Hello World") [04:18:58] Config file is "/data/project/ext-lnk-discover/.lighttpd.conf" [04:20:00] Lighttpd error log is "/data/project/ext-lnk-discover/error-log" [04:20:28] Thanks for any help. [05:37:10] 3Wikimedia Labs / 3deployment-prep (beta): SpecialCite's i18n is still being loaded which is breaking CiteThisPage in BetaLabs - 10https://bugzilla.wikimedia.org/71112 (10James Forrester) 3NEW p:3Unprio s:3normal a:3None Ideally we'd need a way to not load it from production's extension-list… [05:47:38] 3Wikimedia Labs / 3deployment-prep (beta): SpecialCite's i18n is still being loaded which is breaking CiteThisPage in BetaLabs - 10https://bugzilla.wikimedia.org/71112#c1 (10Krinkle) http://en.wikipedia.beta.wmflabs.org/wiki/Special:Cite > Page not found Should redirect to CiteThisPage (as configured by Cit... [12:49:40] 3Wikimedia Labs: fawiki database - 10https://bugzilla.wikimedia.org/71118 (10Arash.pt) 3UNCO p:3Unprio s:3major a:3None all tools and apps in Persian Wikipedia that using "fawiki.labsdb" as db server is down and I can't find "fawiki" database in "fawiki.labsdb" ! [13:10:59] where do I find the apparmor logs in the labs instances ? [13:37:12] tonythomas: I think they just go straight to syslog [13:38:19] andrewbogott: I found something in /var/log/kern.log1 though. [13:38:35] I find exim not able to do a curl request in my labs instance , now debugging why [13:41:28] andrewbogott: do you know what user have the right to call the curl in our labs instance ? I found out that I was trying to post with a user=nobody and group=nogroup, and thinks thats why it fails [13:41:36] by default, its the Debian-exim thoguh [13:42:24] You're just using curl to access an external URL? I'd expect that to work for anyone. [13:44:42] andrewbogott: I just removed my user=nobody and group=nogroup line from the exim transport and it seems to do the post successfully, I think [13:44:48] yay [13:59:05] Betacommand: no report for he.wiki missing images? https://tools.wmflabs.org/betacommand-dev/reports/hewiki_articles_missing_files.txt [14:04:48] andrewbogott: I am able to get this command https://dpaste.de/cUK6 executed while the exim delivery is happening ( asking this shell to execute when the exim recieves a bounce, in the pipe transport ), but when I remove the echo " "> and make something like this https://dpaste.de/yQSu -- exim shows a child process killed error message. [14:07:44] hm, that does sort of sound like app armor. [14:08:06] But, I don't know anything specific [14:08:33] andrewbogott: oh no :( [14:08:47] andrewbogott: who should I be pinging on app armor ? [14:09:17] tonythomas: this doesn't sound like a labs-specific issue, just a generic linux config thing. So… google? [14:11:23] tonythomas: sorry, I really don't know anything -- and I'm in the middle of a different task [14:13:22] andrewbogott: of course. :) I will dig it more [14:20:42] I am receiving 504 Gateway Time-out accessing a new web proxy. If I do an iptables -nvL it shows a DROP policy, that does not exist on other instances in the same project [14:21:04] Chris-J_WMDE: what instance and project? [14:21:19] scrumbugz phab06 [14:22:34] matanya: looking [14:22:41] thanks [14:23:28] heh, your security group sure does have a lot of entries for port 22 [14:24:16] Chris-J_WMDE: are you accessing as http or https? It looks to me like http (22) is not open outside of labs. https://wikitech.wikimedia.org/wiki/Special:NovaSecurityGroup [14:24:33] http is 22?! [14:24:44] oh, wait [14:24:47] matanya: you can blame Coren [14:24:51] * andrewbogott is not awake yet [14:24:58] :) [14:25:05] that is my default Betacommand :D [14:25:09] * matanya blames Coren [14:25:26] Chris-J_WMDE: nm, security groups look ok [14:25:34] matanya: Im getting a "ERROR 1045 (28000): Access denied for user 's51058'@'localhost' " [14:25:37] In this particular case, you /can/ blame me. I pushed a change earlier than I should have. [14:25:41] It's being fixed now. [14:25:51] Coren: :P [14:25:56] we love you Coren [14:26:00] Betacommand: @'localhost' though is a fail on your end. :-) [14:26:14] Coren: why do you say that? [14:26:35] Im sshed into tools [14:26:50] There is no database you should be accessing without a -h directive, in which case it should be @'whereveryouare' :-) [14:27:15] Coren: sql hewiki_p <$HOME/sql_queries/hewiki_articles_missing_files.sql> $HOME/public_html/reports/hewiki_articles_missing_files.txt [14:27:30] Hm. What? [14:27:33] * Coren checks. [14:27:36] same command Ive been using for a whole [14:27:41] *while [14:28:02] Chris-J_WMDE: ok, I've learned nothing. Seems most likely to be something local to the setup of that instance. [14:28:20] Betacommand: There might be something that confuses the sql script. Lemme check. [14:28:30] I can curl http://localhost, but even within labs, I cannot access the instance over 80 [14:29:04] Coren: similar command works for enwiki [14:29:33] Betacommand: Ah; I see what's going on. It's a side effect of the work-in-progress to fix the real problem. It should be back shortly. [14:29:43] Coren: think this may be related to the s6 problems [14:29:53] The sql script is just really unhelpful about figuring out what's happening. [14:30:18] why would I have a different IP table for the new instance than the old ones? [14:31:37] Chris-J_WMDE: Because one of the databases changed underlying IP. But I'm getting rid of the entire iptables requirement today. I'll have an updated hosts file for every instance shortly. [14:31:39] phab06 reports: Chain INPUT (policy DROP 515 packets, 97356 bytes) [14:31:42] matanya: the list should either populate tomorrow or if I get to it after the fix today [14:31:59] thanks a lot Betacommand [14:32:38] matanya: all of my reports are automated and for the most part I dont check on them unless someone complains [14:32:53] this how it should work [14:34:34] matanya: once things are up I log into labs about once a month [14:35:18] me too :) [14:35:43] and my /home is still only about 600Mb :) [14:41:09] 3Wikimedia Labs / 3tools: Block TweetmemeBot UA - 10https://bugzilla.wikimedia.org/71120 (10Betacommand) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier Please block all UAs with TweetmemeBot in it. Its anywhere from 60-90% of my access logs and wastes tool resources. [14:42:54] 3Wikimedia Labs / 3tools: Block TweetmemeBot UA - 10https://bugzilla.wikimedia.org/71120 (10Betacommand) [14:42:54] 3Wikimedia Labs / 3tools: [tracking] Block spider / web crawler on tool labs - 10https://bugzilla.wikimedia.org/68300 (10Betacommand) [15:22:38] matanya: since I no nothing at all about ferm syntax, can you opine on https://gerrit.wikimedia.org/r/#/c/161955/1 [15:23:27] um… s/no/know/ [15:30:34] or… coren? [16:08:44] !log deployment-prep updating HHVM to 3.3.0-20140918+wmf1 [16:08:47] Logged the message, Master [16:43:56] andrewbogott: i'm not sure if ferm srange can handle inline template [16:44:26] matanya: It's used elsewhere in our codebase, so I think it'll work... [16:44:39] so you code sure work as well [16:44:43] *your [17:06:53] 3Wikimedia Labs / 3tools: Grid engine: fail to submit job - 10https://bugzilla.wikimedia.org/71125#c1 (10Marc A. Pelletier) 5NEW>3RESO/FIX Job spool ran out of space. Fixed. [17:11:37] Change on 12mediawiki a page OAuth/For Developers was modified, changed by Ragesoss link https://www.mediawiki.org/w/index.php?diff=1178757 edit summary: /* Golang demo cli client with HMAC */ link to the MediaWiki OmniAuth strategy that is available [17:16:20] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [17:35:53] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Views completely missing from some wiki's - 10https://bugzilla.wikimedia.org/71041#c9 (10Marc A. Pelletier) This should be fixed right now; with Sean having merged all three of the labsdb1003 slices and the hosts file adjusted accordingly. [18:07:06] I have problems logging [18:07:17] via putty [18:07:36] in tools-login.wmflabs.org [18:11:54] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Views completely missing from some wiki's - 10https://bugzilla.wikimedia.org/71041#c10 (10metatron) Some user databases still missing. See #3. [18:15:22] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Wiki ViewStats databases completely missing! - 10https://bugzilla.wikimedia.org/71043#c1 (10metatron) Added another listener for test, worked fine until 2014-09-22 05:39:56. Now: ERROR 1290 (HY000) at line 1: The MariaDB server is running with the --read-onl... [18:23:10] magioladitis: what's the error? [18:24:52] !log extdist manually restarted nginx [18:24:56] Logged the message, Master [18:25:00] valhallasw`cloud: Disconnected: No supported authentication methods available [18:25:10] magioladitis: set up your private key [18:25:20] hrmm, is account creation on wikitech broken atm? it reports failure for a perfectly valid shell account name [18:25:21] valhallasw`cloud: I did. Where do I put this [18:25:39] magioladitis: see https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP [18:26:08] specifically https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP#mediaviewer/File:20130526_2133_Putty_Login_Connection_SSH_Auth.png [18:26:43] valhallasw`cloud: thanks. I have to leave home for a couple of hours. I ll try it. If I still have problems "I 'll be back" [18:26:59] andrewbogott / Coren / yuvipanda ^^ my message above [18:27:20] (onboarding a new dev:) [18:27:22] MaxSem: maybe the name is already taken? [18:27:26] MaxSem: Not that I know of; what account name are you trying? [18:27:35] bmansurov [18:27:57] wasn't on ListUsers [18:27:59] Doesn't already exist so that's not that. [18:28:25] (All lower case, right)? [18:28:34] yup [18:28:52] I reproduced it myself after he reported a problem [18:29:56] MaxSem: I'll be taking a look at the logs after the ops meeting then. [18:30:03] thanks [18:34:04] MaxSem: Actually, make you make an attempt right now? [18:34:53] Coren, just did [18:36:59] Hmm. Odd. What is the wiki name and shell acount you are trying? "BMansurov / bmansurov"? [18:37:14] Oh, good [18:37:21] Someone is in #mediawiki reporting the same issue [18:37:24] yup [18:38:00] "There was either an authentication database error or you are not allowed to update your external account." [18:38:17] Username Sushmareddy05, shell name sushmareddy [18:38:36] Bleh. The logs are silent. [18:38:58] ldap being a jerk? [18:40:01] lderp [18:40:19] Hii [18:40:27] I know andrewbogott is messing with ldap right now, but I don't think in a way that should affect virt1000 [18:40:41] Is anyone here? [18:40:41] Hi sushmareddy, I am here, if you need anything, please ask, otherwise no one is going to help you... Thank you [18:40:59] Coren: vaguely possible, but ldap seems to be working otherwise [18:40:59] Yeah...I need a help [18:41:20] sushmareddy: The channel already knows about your issues, you can safely lurk :) [18:41:21] Coren, it's "bmansurov / bmansurov" [18:41:24] sushmareddy, having problems creating an account? people are on it [18:41:25] (03CR) 10Ori.livneh: [C: 032] Add password for puppet_compiler [labs/private] - 10https://gerrit.wikimedia.org/r/159010 (owner: 10Ori.livneh) [18:41:54] Ok thanks a lot marktraceur :) [18:42:09] andrewbogott: Well, I can read from it fine. Does opendj keeps logs anywhere? [18:42:18] Yeah can you help me please [18:42:34] sushmareddy: We're working on it. [18:42:52] Ok! :) [18:42:57] Coren, is ldaplist broken as well? [18:43:39] Nope. That works for me. [18:44:06] * MaxSem goes to grab some food meanwhile [18:44:30] Coren: ldaplist -l passwd? [18:45:28] Hm. "ldaplist -l passwd " works; "ldaplist -l" doesn't. [18:45:28] oh, I bet we just crossed a threshold of too many users since that worked a few days ago... [18:45:39] Ah! Index limit! [18:46:05] Coren: I'm having a hard time thinking about this and being in a meeting and having a migraine all at once. But… yes, opendj logs in /var/opendj/instance/logs [18:46:13] and also I can turn on auth logging for wikitech, one moment... [18:47:39] So… I'm editing /srv/mediawiki/wmf-config/wikitech.php, uncommenting a couple of debug lines... [18:48:55] Coren: so, now, more logging in /tmp/ldap-s-1-debug.log [18:49:41] andrewbogott: "[novaadmin does] not have sufficient privileges to perform an unindexed search" on ou=people [18:50:04] Yep, we hit a lucky number. We should have a party! [18:50:31] I don't know how to allow unindexed searches… hopefully this is possible without giving that account too many other privs [18:51:24] andrewbogott: The alternative is to just index the thing. [18:51:39] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Wiki ViewStats databases completely missing! - 10https://bugzilla.wikimedia.org/71043#c2 (10metatron) Documenting some things. (though supposedly re-attached & inactive) Mon Sep 22 18:49:01 UTC 2014 | Database | Table | In_use | +... [18:52:34] Coren: looks like there is a specific flag for that, probably can just enable big searches [18:59:37] Can you fiddle the flag? [18:59:53] Coren: language! [19:00:04] this is a family-friendly channel [19:01:33] * MaxSem swears dirtily: MediaWiki, LDAP, Swift containers, Jenkins [19:01:50] Python! [19:03:43] Coren: yes the language that works and is readable [19:07:50] Coren and/or MaxSem, try now? [19:08:10] bmansurov, ^ [19:08:28] Coren: Any comments on https://bugzilla.wikimedia.org/show_bug.cgi?id=71043 ? Hm? I'd appreciate that... [19:08:48] andrewbogott: That fixed ldaplist at least. [19:09:41] I'm aware, hedonil. It should be fixed sometime today. [19:10:38] Coren: wow, wow [19:10:53] wow? :-) [19:11:56] Coren: wow! ;-) [19:11:59] WOW. [19:12:08] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Wiki ViewStats databases completely missing! - 10https://bugzilla.wikimedia.org/71043#c3 (10Marc A. Pelletier) 5NEW>3ASSI The issue is known, and should correct itself once the database merge is complete. [19:13:27] * hedonil becomes a *little* bit sarcastic, but is still full of hope... [19:14:25] 3Wikimedia Labs / 3Infrastructure: cn=proxyagent,ou=profile,dc=wikimedia,dc=org should probably not allow unindexed searches - 10https://bugzilla.wikimedia.org/71130 (10Andrew Bogott) 3NEW p:3Unprio s:3normal a:3None Right now we depend on unindexed searches to make some ldap tools work, e.g. ldaplis... [19:14:36] !log tools.wikiviewstats contact! [19:14:39] Logged the message, Master [19:15:22] Coren: just a heads up the sql query I was running earlier is now working again [19:16:19] Betacommand: Yes, the actual replicas themselves should no longer pose issues; what's left is some of the user dbs. [19:17:03] Coren: I remember the days when the databases had 1+ years of lag :P [19:19:42] MaxSem, thanks [19:21:37] !log tools.wikiviewstats Stardate -308274.086757991 Cdr Hedonil speaking. First contact about bug 71043. No clue about the cause so far, but hings could run off lightly. We'll stay tuned. [19:21:39] Logged the message, Master [19:33:56] bmansurov: Did it work, then? [19:55:11] Coren, it did [19:55:22] sushmareddy: That means it should work for you, too. [20:20:11] Coren, yes it did [22:10:27] Is there any known problem with life time of php sessions on labs tools? [22:13:56] May I set session.save_path manually? [22:24:43] Well I tried setting session.gc_maxlifetime, session.cookie_lifetime, session_set_cookie_params it gets better but after one hour at seems sessions are removed [23:05:25] Error: mwscript importDump.php --wiki=wiki /vagrant/puppet/modules/labs/files/labs_privacy_policy.xml returned 1 instead of one of [0] [23:05:25] Error: /Stage[main]/Role::Labs_initial_content/Mediawiki::Import_dump[labs_privacy]/Exec[import_dump_labs_privacy]/returns: change from notrun to 0 failed: mwscript importDump.php --wiki=wiki /vagrant/puppet/modules/labs/files/labs_privacy_policy.xml returned 1 instead of one of [0] [23:07:05] bd808: so http://gadgets.wmflabs.org/ links to itself... [23:07:14] bd808: I think I need to set something in heira ? [23:07:53] legoktm: A patch for the import dump bug was merged just a few minutes ago. [23:08:24] legoktm: for the other bit, you probably do need some hiera hacking. [23:08:36] What wiki farm are you trying to setup? [23:08:50] are you referring to https://gerrit.wikimedia.org/r/#/c/161140/ ? my labs-vagrant already has that commit [23:08:57] gadgets2 role [23:09:52] bd808: i think it will still fail without the follow-up patch [23:10:02] legoktm: Ugh. If 161140 didn't fix your bug then I know the cause. I'm working on a patch for it in ops/puppet [23:10:22] "your bug" == importDump.php failing [23:10:31] ok, do you know how to fix the wiki not loading part? [23:10:39] http://gadgets.wmflabs.org/ says "No wiki found" [23:10:47] and the link to "devwiki" is back to itself [23:11:02] It is probably related. There is a permissions problem in the /vargrant dir [23:11:25] This is what I'm working on the other puppet patch for [23:11:27] ok [23:11:36] any quick hack I can try? :P [23:12:04] you can "fix" with `sudo chmod -R o+rX /srv/vagrant` until the next time puppet runs [23:12:19] I think that will fix it anyway [23:12:29] Maybe you can confirm [23:13:19] There's a bit of runaway code in something Ori and I hacked up for scap at some point and it's biting labs_vagrant now. [23:13:45] * legoktm tries [23:14:03] yay! [23:15:17] bd808: aaand now the wiki loads. yay thanks! [23:15:54] I'll try to con someone into merging the real fix, but ops/puppet so... [23:27:17] bd808: umm, how do I turn off puppet? [23:27:50] sudo puppet agent --disable 'puppet is breaking my wiki' [23:28:11] That looks like a joke but it's for reals [23:28:13] ty :) [23:28:27] !log social-tools turned off puppet on gadgets instance because it was breaking the wiki [23:28:30] Logged the message, Master [23:31:33] legoktm: I think https://gerrit.wikimedia.org/r/#/c/162160/ will make puppet stop breaking things but everyone who has been effected will need a to fix with the recursive chmod :(