[00:14:01] Coren: Group permissions not being applied correctly, though my problem was with service users [00:14:27] so, who should i talk to if i get a 502 proxy error on the page creation page? [02:53:05] Nemo_bis: https://meta.wikimedia.org/wiki/Talk:Privacy_policy#Google_Analytics.2C_GitHub_ribbon.2C_Facebook_like_button.2C_etc. [02:57:30] https://meta.wikimedia.org/wiki/User_talk:Valhallasw#tsreports [03:01:17] Gloria: Does the policy say anything about admins doing so on public wikis via Common.js/css? [03:03:49] https://meta.wikimedia.org/wiki/Talk:Privacy_policy/Archives/2013#Exclusion_of_on-wiki_actions_from_privacy_policy [03:04:00] I'm not sure what the final answer was. [03:05:20] Duno. [03:05:21] Dunno. [03:05:23] Dunnno. [04:04:59] ? [04:06:35] . [04:07:25] Beetstra: ! [08:59:03] Change on 12mediawiki a page Wikimedia Labs/Tool Labs/Migration of Toolserver tools was modified, changed by Nemo bis link https://www.mediawiki.org/w/index.php?diff=858539 edit summary: /* What do I have to think of when finishing migration? */ update [10:59:01] Hello All! Why do I get "Internal Server Error" on all my cgi scripts today? Somebody else having same issues? [11:07:10] Coren: ^^^ [11:14:29] Anybody here?! :) [11:17:08] giftpflanze: Are you here? [11:17:57] we might be there DrTrigon :) [11:18:30] saper: Next I woul have ping-ed you... ;) [11:19:04] saper: Do you have an idea, why do I get "Internal Server Error" on all my cgi scripts today? Somebody else having same issues? [11:24:07] saper: ??? [11:25:17] what do logs say? [11:25:37] saper: which ones? [11:29:51] where is there problem? on the "new" toolserver? [11:32:31] saper: e.g. this script http://tools.wmflabs.org/drtrigonbot/cgi-bin/filter.py [11:33:57] Error logs, because of limitations of the Apache web server, are not made directly available to tool maintainers. Until a newer version of Apache is deployed, we recommend that you use your language's facilities to log errors to a file under the tool account's home. PHP allows per-user logging, for example, and PHP error logs are placed in the tool account’s ~/php_error.log. [11:34:03] out of luck [11:34:28] is any basic CGI working? like "#! /bin/sh \n env" ? [11:35:23] saper: It worked yesterday and before - but not today - I changed nothing meanwhile (as usual ;) - and it affects ALL my scripts [11:35:58] saper: I am using cgi module for debug, but it does not appear here, so I assume the script does not get executed at all... [11:37:06] maybe something changed with python [11:37:24] is the simples shell CGI working? [11:37:26] or permissions, or else... [11:37:40] saper: moment please... [11:40:51] saper: no! http://tools.wmflabs.org/drtrigonbot/cgi-bin/test (/data/project/drtrigonbot/cgi-bin/test) [11:41:19] some .htaccess? [11:41:24] not sure it's allowed [11:41:52] saper: not change from my side... [11:43:12] The requested URL /saper/test.html was not found on this server. [11:43:12] Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request. [11:43:15] :) [11:43:28] exactly! [11:44:21] eveven out of luck [11:44:29] (there is even an error while error handling...) [11:44:40] saper: so what to do now?? [11:45:23] not sure; bugzilla? :) [11:45:52] saper: what component, etc. what exact issue...? [11:46:00] no idea, sir [11:46:17] I am still using mostly the old, deprecated toolserver [11:46:58] saper: me too - but still I would like to test here... [11:47:02] ;)) [11:48:51] saper: could you give me the exact link to your 'test.html' for bug report please? [11:56:13] DrTrigon: yeah, only plain HTML works [11:56:23] so test.html works now [11:57:55] saper: do you have another link example which does NOT work at the moment? [11:58:08] http://tools.wmflabs.org/saper/cgi-bin/simple [11:58:14] but I created it just right now [11:58:24] it's /bin/sh script [11:59:13] cool! sorry I meant html... ;)) [11:59:37] no, HTML works [14:59:19] !log bots booted wm-bot again [14:59:19] Logged the message, Master [15:56:10] pyexiv2 is on the login node but not onthe worker nodes [15:56:14] ImportError: No module named pyexiv2 [15:58:53] dschwen: Someone must have installed it manually. I don't recall it ever having been in a bugzilla. [16:03:03] yeah [16:03:05] i know [16:03:13] first thing I did is a bugzilla searcg [17:22:39] coren, up? Thinking about https://bugzilla.wikimedia.org/show_bug.cgi?id=58997 again [17:33:13] are there known problems with the toolserver? six hours ago I only got 500 (like reported by DrTrigon), now I'm not even able to get any response from any tool incl. tools.wmflabs.org (the main page) [17:34:58] downforeveryoneorjustme.com tells me: It's not just you! http://tools.wmflabs.org looks down from here. [17:35:35] I meant "Tool Labs" instead of "Toolserver", of course [17:42:27] apper: WFM [17:42:46] first load of http://tools.wmflabs.org/ was a bit slow though [17:42:55] maybe needs some caching or something [17:43:33] is it possible to ping tools.wmflabs.org normally? [17:43:37] So… not broken? [17:43:43] this doesn't work for me, either [17:44:20] Hm, I can ping [17:44:41] hmmm [17:45:51] can you ping wikitech.wikimedia.org? [17:46:02] okay, I'm able to ping from toolserver and from my university, but not from my home PC [17:46:14] andrewbogott: yes [17:46:20] Curious! [17:46:50] apper: how long do you leave the ping running? [17:48:08] ping is sometimes taking time to return results but also not losing packets and not saying high latency... [17:48:11] 10 packets transmitted, 10 received, 0% packet loss, time 45415ms [17:48:12] rtt min/avg/max/mdev = 41.893/42.196/42.836/0.309 ms [17:48:46] 11 149 ms 149 ms 149 ms xe-0-0-1.cr1-sdtpa.wikimedia.org [208.80.154.210] [17:48:46] 12 150 ms 150 ms 150 ms 208.80.153.201 [17:48:46] 13 208.80.153.201 meldet: Zielprotokoll nicht erreichbar. [17:48:52] that's what traceroute tells me [17:48:59] right, don't use traceroute [17:49:08] okay [17:49:11] see my paste. you have to wait a long time [17:49:15] (which is wrong) [17:49:27] 45 secs for 10 pings @ -i 1. well under 1 sec response time. idk what's wrong [17:49:55] compare with pinging 8.8.8.8: [17:49:56] 10 packets transmitted, 10 received, 0% packet loss, time 8995ms [17:49:56] rtt min/avg/max/mdev = 17.276/22.271/48.199/9.927 ms [17:51:23] hm, maybe this is a thing with the windows ping... but nevertheless, wmflabs.org is not available for me [17:52:01] ping tells me, that the target port is not available... but only for tools.wmflabs.org [17:52:22] btw bots.wmflabs.org works [17:52:53] apper, I need to run… if this turns out to somehow be a real problem rather than with your local network, ping me on my return :) [17:54:28] I was told there were a lot of accesses to several tools from the german provider QSC today (someone switched one of my tools to the new webserver stuff because of it). I'm also using QSC - is it possible, that access from certain IPs is blocked? [18:00:07] apper: do you have anything besides windows there? [18:01:38] to summarize: I'm able to access everything (including tools-login.wmflabs.org), and I'm also able to access tools.wmflabs.org from other servers - but not from my PC... that's weird [18:01:56] jeremyb: nope. I can test two other windows PCs... [18:02:50] apper: i suppose you could just `ping tools.wmflabs.org`, wait half a min, ctrl-c [18:03:19] I can't reproduce from a colo box but i have the issue as posted above from my connection [18:03:24] bbl [18:04:55] same problem from two other PCs (both windows) in the same network [18:09:07] it seems, that it's not about the time... I'm getting answers from ping, but it seems to be a "Destination unreachable" message, see http://en.wikipedia.org/wiki/ICMP_Destination_Unreachable#Destination_unreachable [18:09:59] seems to be "Code: 3 (Port unreachable error)" [18:12:16] is it possible to exclude the possibility, that a certain IP range is blocked? [21:44:41] Coren: So, regarding https://bugzilla.wikimedia.org/show_bug.cgi?id=58997 -- I think the service user namechange thing ruins our scheme to use both schema at the same time. Since a user can only have one name... [21:45:24] Why? They'll have only one at a time; /which/ just depends on what OU we point the LDAP clients to. Just maintain both entries. [21:45:32] They'd be "different" users. [21:45:43] That "happen" to have the same UID etc. :-) [22:41:28] Coren: have you read my messages above? I'm not able to connect to tools.wmflabs.org, it seems connections from my IP are blocked (it works from any other PC and even from the same laptop in a different network), and it only affects tools.wmflabs.org, all other sites work. I was told there were problems with a lot of requests from a german provider (QSC) today, so maybe an IP range was blocked? do you know something about this? is it [22:41:29] possible to block all incoming connections from a specific IP range? If so, how can I see if I'm affected? [22:45:00] apper: i'm sorry, you should provide more info and i'm unwilling to figure out how you should do that on windows [22:45:14] on linux or mac i would say `nc -v tools.wmflabs.org 22` [22:45:44] good luck! [22:47:48] there is a windows version of netcat [22:47:52] Warning: inverse host lookup failed for 208.80.153.201: h_errno 11004: NO_DATA [22:47:52] tools.wmflabs.org [208.80.153.201] 22 (ssh): TIMEDOUT [22:49:50] apper: I can check if you are affected if you PM me your current public IP [22:51:27] apper: You weren't caught in a range; your IP was specifically blocked as one of the top 5. :-) [22:51:44] apper: Lemme look at the logs. [22:51:48] do you know when it was blocked? [22:52:19] (and when it was in the last days: why?) [22:53:26] <^d> Um, if I have 2fa enabled on my wikitech account, but I had to wipe my phone and lost my authenticator data, how do I get back in? [22:53:49] Around 13:10 UTC today; the reason is simple, I see dozens of requests/second for /ipp/npp_extern.php?threshold=20&thresholdipp=0.98, [22:54:10] ah [22:54:11] okay [22:54:44] ^d: You use the reset token you wrote down when setting up the 2fa, that'll allow you to rekey your phone. [22:55:18] cue Ryan [22:55:25] apper: Give me a minute to remove the block [22:55:27] Coren: thanks, good to know. That was me, indeed (in particular: a program running crazy) [22:55:34] <^d> Where the hell did I write that down? [22:55:37] * ^d thinks [22:56:13] ^d: The alternative is a manual reset of your credentials in the DB, and that will require out of band authentication. [22:56:36] Coren: the program wants a persistant connection and tries it over and over if it gets a 500 for example - and that was the case this morning. I will just change the program, that this doesn't happen again [22:56:58] <^d> Coren: We might have to do that...I can't for the life of me remember where I recorded that reset key. [22:57:00] apper: Trying again is okay; just not that often or that fast. :-) [22:58:26] ^d: Are you physically at the office atm? [22:59:08] apper: Unblocked. [22:59:16] ^d: Ryan was just complaining about you people 9 days ago... [22:59:25] <^d> Coren: I am not, I am at home waiting on a UPS delivery. [23:00:17] Coren: that happens when hacking together a simple script and you're not aware of possible problems at the server side ;). I was just catching the error and tried again. I now wait some time before trying again. Thanks for helping [23:01:00] apper: Best practice: https://en.wikipedia.org/wiki/Exponential_backoff [23:01:28] <^d> Coren: I fail, sorry :( [23:01:39] Coren: thanks [23:02:33] ^d: Pick up your phone. :-) [23:03:13] <^d> I was like who the heck is calling me? :p [23:06:05] * Coren now tries to remember where in the database that lives. [23:06:30] High security question: "Are you you?" [23:12:12] when using the new webservice with "webservice start" - is this like starting the service with the continuous option? So unless I don't stop it manually it will run forever? Is there a memory limit? [23:13:07] apper: Yes, but it's bigger than you need. [23:13:41] (And, in practice, the memory footprint isn't all that big since the only things running on those nodes are the same daemons) [23:14:00] okay, thanks [23:15:08] ^d: The bad news is: Ryan's notes on how do remove 2fa from an account were not sufficient for me to figure it out easily. It'll take a bit of RTFS to figure it out. [23:15:55] <^d> I'm not in any particular rush if you've got other things to get to [23:16:27] ^d: In practice, you're without login until I do it so... :-) I better find it now than procrastinate. [23:28:30] Coren: what docs did you find? [23:29:50] Coren: https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FOATHAuth/534beaf0f38780be4ad6b6f11763cbfcded2bffa/OATHUser.php#L275 [23:37:19] jeremyb: That's what I was looking at, but according to this it'd be oath-* in preferences and I can't find it there. [23:37:30] Or, I could just ask Ryan_Lane who conveniently just arrived. [23:37:45] Ryan_Lane: Where in the DB is the 2fa setup? [23:38:08] virt0's mediawiki db [23:38:22] oathauth table, or something like that [23:38:33] the id in that table is the user's id from the user table [23:39:22] ... [23:39:40] ? [23:39:41] That's what I was looking for. It'd have helped if I didn't mindlessly look in the /wrong/ db. [23:42:41] Nevermind me. :-) [23:43:15] heh. all these project deletions and I'm still in way too many [23:43:24] login takes a while for me. I bet it needs to do a ton of ldap lookups [23:44:00] I'll need to take a look into domain settings in keystone in havana [23:44:11] may be able to set up domain groups [23:44:53] ^d: You've been de-2fa'ed. [23:45:27] <^d> Thank you! [23:47:56] I'm definitely going to add a check in for activating 2fa to enter a scratch token [23:48:05] and I'm going to put it on a separate challenge screen [23:48:14] to ensure people have written them down [23:50:40] <^d> Ryan_Lane: I did write it down. I can't remember where the hell I did. [23:50:48] :D