[00:06:09] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1426876 (10Krenair) This also prevents us from being able to add silver to the wikis which get QueryPages like Special:Wantedpages automatically updated [01:28:46] 6Labs, 10VisualEditor, 10VisualEditor-MediaWiki, 10wikitech.wikimedia.org, 7Regression: Pages can't be saved with VisualEditor on wikitech (Uncaught ReferenceError: attrId is not defined) - https://phabricator.wikimedia.org/T104360#1426888 (10Krinkle) 5Open>3Resolved a:3Krinkle >>! In T104360#14258... [01:29:31] 6Labs, 6Analytics-Backlog, 10wikitech.wikimedia.org, 7Varnish: https://wikitech.wikimedia.org/beacon/statsv 404 Not Found - https://phabricator.wikimedia.org/T104359#1426895 (10Krinkle) [01:31:24] 6Labs, 6Analytics-Backlog, 10wikitech.wikimedia.org, 7Varnish: https://wikitech.wikimedia.org/beacon/statsv 404 Not Found - https://phabricator.wikimedia.org/T104359#1426898 (10Krinkle) Related: {069716447f35c4aee6} > https://gerrit.wikimedia.org/r/221969 Though that didn't fix it. [02:07:10] hi, I'm getting this at Special:NovaInstance, is this normal: https://usercontent.irccloud-cdn.com/file/masJ7unk/Screen%20Shot%202015-07-04%20at%209.06.02%20PM.png [02:07:21] I can't click any of the dropdowns [02:09:48] try logging out and back in again [02:40:54] any luck L235? [02:41:14] oh, sorry, yeah, it worked [02:44:28] wikitech is weird - sometimes it says you're logged in but parts of it don't know about it and produce weird errors. Logging out/in always fixes these for me. [02:55:52] https://phabricator.wikimedia.org/T101199 [02:56:27] makes it seem like the problem is related to how the OpenStackManager extension handles the internal OpenStack/Nova session [04:09:55] 10Tool-Labs-tools-Other, 6WMF-Legal: Another request to review privacy policy and rules - https://phabricator.wikimedia.org/T104784#1426962 (10Ricordisamoa) 3NEW [04:11:14] 10Tool-Labs-tools-Other, 6WMF-Legal: Another request to review privacy policy and rules - https://phabricator.wikimedia.org/T104784#1426969 (10Ricordisamoa) [04:15:56] 10Tool-Labs-tools-Other, 6WMF-Legal: Another request to review privacy policy and rules - https://phabricator.wikimedia.org/T104784#1426972 (10Ricordisamoa) [04:20:19] 10Tool-Labs-tools-Other, 6WMF-Legal: Another request to review privacy policy and rules - https://phabricator.wikimedia.org/T104784#1426977 (10Ricordisamoa) [06:18:56] Has anyone gotten cherrypy to work? [06:27:27] 6Labs, 10Tool-Labs: Install composer on tools-bation - https://phabricator.wikimedia.org/T104789#1427031 (10Krinkle) 3NEW [06:58:29] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL 50.00% of data above the critical threshold [0.0] [07:15:39] [13intuition] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/4c9520615cd0f98941065e921532585c39061389 [07:15:40] 13intuition/06master 144c95206 15Timo Tijhof: Delete LocalConfig-wmflabs.php... [07:23:32] RECOVERY - Puppet failure on tools-exec-gift is OK Less than 1.00% above the threshold [0.0] [07:34:40] 10Quarry, 5Patch-For-Review: Quarry does not respect ORDER BY sort order in result set - https://phabricator.wikimedia.org/T87829#1427061 (10MahmoudHashemi) Awesome dude. Appears to be working on my end from the queries I tried. And hey, believe me, I know a thing or two about software development processes an... [07:38:20] [13intuition] 15Krinkle pushed 2 new commits to 06master: 02https://github.com/Krinkle/intuition/compare/4c9520615cd0...1dcb4fdbc889 [07:38:20] 13intuition/06master 147522a9b 15Timo Tijhof: api: Restore language fallback support in API responses... [07:38:21] 13intuition/06master 141dcb4fd 15Timo Tijhof: build: Fix npm-test warning about missing license... [09:17:01] o/ anyone awake? [10:26:33] Is the http://mathoid.testme.wmflabs.org/ unavailability related to the NFS issue? [10:30:42] Celelibi: unlikely, but it's a possibility. Check with the mathoid people? [10:31:00] How? Where? [10:31:33] Celelibi: I don't know? The person who gave you that link? [10:31:57] This extension: https://www.mediawiki.org/wiki/Extension:Math [10:32:17] It's even on that page. [10:33:16] Celelibi: ...in the example configuration [10:34:25] valhallasw`cloud: It's the default value in the distributed extension. [10:37:36] If the server still exists -- I'm not sure whether it does, as it could also be a placeholder vlaue -- it's probably run by https://wikitech.wikimedia.org/wiki/Nova_Resource:Math [10:41:56] Well, it used to work. [10:56:59] 6Labs, 10Tool-Labs: Install composer on tools-bation - https://phabricator.wikimedia.org/T104789#1427199 (10yuvipanda) Is there a Debian package for it? [11:03:44] 6Labs, 10Tool-Labs: Install composer on tools-bation - https://phabricator.wikimedia.org/T104789#1427206 (10valhallasw) There's an package in debian-unstable: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=714118 [11:04:25] 10Quarry, 5Patch-For-Review: Quarry does not respect ORDER BY sort order in result set - https://phabricator.wikimedia.org/T87829#1427207 (10yuvipanda) :) part of the problem was the lack of a local dev environment - I just fixed that problem yesterday (vagrant!) - fixing things should be easier now :) [11:08:34] question of anyones awake [11:08:56] I'm having trouble with ssh keys - I validated they work thanks to -A (and intend on rotating them once I resolve the issue) [11:09:01] but don't know what's making things sad [11:10:06] elee: just ask the question, if anyone has a solution, they'll answer [11:10:38] YuviPanda: it might be hard to deploy a debian-unstable deb, though :( [11:14:55] YuviPanda: dpkg -i fails in any case due to dependencies [11:15:17] Ugh sigh [11:15:33] Might need backported / rebuilding [11:18:19] YuviPanda: or we can just curl | sudo php! [/jk] [11:18:41] :p [11:18:52] I wonder how extdist does it [11:19:01] It uses composer [11:22:34] YuviPanda or valhallasw`cloud, may I see your .ssh/config? [11:22:48] this is mine, which I essentially copy pastaed from somewhere from wikitech: https://github.com/leee/.dotfiles/blob/master/sshconfig [11:22:54] but I guess... it doesn't work? =/ [11:23:26] elee: I don't know why you have all those prod Hosts there, but otherwise looks ok? [11:23:31] I'm still in bed and not of much use sorry [11:23:34] hrm =/ [11:23:35] what's the actual issue? [11:23:40] YuviPanda: its alright, sleep is good [11:24:19] I get perm denied because of my pub key [11:24:31] (and therefore get killed by sig 1) [11:25:05] what host are you connecting to, and what's the output of ssh -vv [11:26:14] oh hrm this is interesting [11:26:16] standby [11:27:58] web.mit.edu/leee/Public/ssh-stawp.txt [11:28:10] uh yeah apologies in advance about the machine name [11:29:14] elee: ok, so the connection to bastion works, but then the connection to xtools-a fails [11:29:21] yeah, and I'm going hrm [11:29:31] so when I eval `ssh-agent` and toss it my keys [11:29:42] and go into bastion, then ssh -A from there, it works [11:30:12] wait [11:30:15] debug1: Trying private key: /Users/leee/.ssh/wm-labs [11:30:15] debug1: key_parse_private_pem: PEM_read_PrivateKey failed [11:30:30] that's... weird [11:30:33] oh wait hrm [11:30:40] is bastion, not [11:30:52] but then it does seem to work with Identity added: /Users/leee/.ssh/wm-labs (/Users/leee/.ssh/wm-labs) [11:31:00] no, you just use bastion to create a tunnel [11:31:22] right which is why this seems unordinary [11:31:50] could you try ssh -vv bastion.wmflabs.org, and compare the auth handshake? [11:33:39] cursory look checks out: http://web.mit.edu/leee/Public/ssh-stawp2.txt [11:34:11] wait nevermind [11:34:27] near the end: debug1: key_parse_private_pem: PEM_read_PrivateKey failed [11:34:28] ugh [11:34:52] yeah, that's the same, but it still works [11:34:53] but [11:34:56] debug2: we sent a publickey packet, wait for reply [11:34:57] vs [11:35:11] err [11:35:19] debug2: we sent a publickey packet, wait for reply [11:35:19] debug1: Authentications that can continue: publickey [11:35:19] debug2: we did not send a packet, disable method [11:38:02] I'm at a loss here. [11:38:30] last thing you could compare with is ssh -vv xtools-a while you're on bastion with agent forwarding on [11:41:59] labs down or something? [11:42:35] comets: tool labs or labs? both seem happy right now [11:42:46] valhallasw`cloud: yeah I'll probably do that after I grab breakfast [11:43:26] well my favorite tool refuses to work today :( [11:43:52] comets: which one? [11:44:15] croptool [11:44:41] https://tools.wmflabs.org/croptool/ [11:46:02] working for you? if not, can someone with superdubahpowahs, reboot it? [11:50:12] 6Labs, 7Mobile: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1427227 (10yuvipanda) Am rebuilding this on the android-builder machine, currently stuck at: ```* What went wrong: Execution failed for task ':wikipedia:proguardAlphaRelease'. > java.io.FileNotFoundExc... [11:50:46] 2015-07-05 11:48:57: (server.c.1398) [note] sockets disabled, connection limit reached [11:51:30] but there's nothing that recent in access.log. Hm. [11:54:38] 10Quarry: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#1427229 (10yuvipanda) @valhallasw how does this translate to quarry? [12:00:29] 10Quarry, 5Patch-For-Review: Return to the starting Quarry page after login - https://phabricator.wikimedia.org/T100162#1427240 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Fixed! [12:11:25] 10Quarry, 5Patch-For-Review: Make the query description window expanable - https://phabricator.wikimedia.org/T77945#1427250 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done and deployed! Thank you for reporting this and apologies for the delayed fixing. [12:11:59] 10Quarry: Quarry shows old username after user was renamed - https://phabricator.wikimedia.org/T73064#1427253 (10yuvipanda) Hmm, good question. I'm not particularly sure how to do this... [12:12:10] 10Quarry: Make it possible to link to latest resultset for a query - https://phabricator.wikimedia.org/T75883#1427254 (10yuvipanda) p:5Triage>3Normal [12:14:19] well? [12:14:47] YuviPanda, can u restart croptool pweaze? :D [12:14:55] its not loading :/ [12:15:04] no [12:15:07] I'm working on it [12:15:39] I think I have the info I need, but I need to check on the proxy [12:17:26] so sometime next week yah? :/ [12:18:42] I'm terribly sorry for putting effort into determining the root cause of this issue, so it could be prevented a next time. I won't do it again. [12:19:25] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: croptools down - https://phabricator.wikimedia.org/T104799#1427257 (10valhallasw) 3NEW [12:21:26] yeah, everytime one guy says they will fix it, takes them days and its still broken..reboot works better. [12:23:06] comets: please behave more respectfully to other volunteers who are putting in time because they care. [12:25:52] yeha its one tool that can never get fixed, i have to ask here all the time for someone to reboot it so i doubt he can fix it..but he can try if he likes. [12:26:57] well, you can continue to be snarky and be ignored. have fun. [12:28:30] well i'm used to being ignored..so no biggie. [12:33:30] 10Quarry, 5Patch-For-Review: Make it possible to link to latest resultset for a query - https://phabricator.wikimedia.org/T75883#1427273 (10yuvipanda) 5Open>3Resolved a:3yuvipanda http://quarry.wmflabs.org/query/2157/result/latest/0/json w00t :) [12:35:20] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Other: croptools down - https://phabricator.wikimedia.org/T104799#1427279 (10valhallasw) The nginx/webproxy log doesn't help too much. I think the dates/times mentioned above might just be wrong (I based the find on https://superuser.com/questions/565991/how-to-determine... [12:39:15] YuviPanda: I think it's some kind of issue with lighttpd not closing the connection and then hanging, but I'm not sure why. [12:39:35] maybe it's a huge thing that just gets the php-cgi thing killed? [12:39:44] also I just realized we could offer people php-fpm, should probably be better [12:40:03] YuviPanda: nah, the server says 'I don't have file descriptors left, so I'm stopping listening' [12:40:17] right [12:40:35] there's probably a limit on 300 connections? [12:40:53] yeah. [12:41:02] I guess one 'solution' is to implement the http endpoint checker... [12:44:41] i doubt 300 ppl use it at once [12:45:08] but it is used as a gadget which may be enabled by more than 300 users on commons.. [13:06:27] 10Quarry: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#1427324 (10valhallasw) I think allowing users to do a LIKE '%xx%' on each column would be the nicest, but I'm not sure if that's doable without killing a database server in the process. [13:21:30] 10Quarry: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#1427330 (10yuvipanda) We can do client side filtering with the data tables library that is already being used? [13:26:48] 10Quarry: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#1427334 (10valhallasw) If that works with pagination it sounds like a good option. [13:31:21] I'm going to ask a stupid question - I can't `become xtools` even though I'm a maintainer - any debugging ideas? [13:33:28] elee: where are you executing that? you're not logged in to tools-bastion-01 [13:33:36] 02 [14:06:36] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1427347 (10JanZerebecki) 5Resolved>3Open Clearing cookies and doing a login results in either a "You have cookies disabled." or being loged out again after reloading. Serv... [14:08:12] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech; works after deleting the cookies - https://phabricator.wikimedia.org/T103939#1427349 (10JanZerebecki) [14:09:19] 6Labs, 10wikitech.wikimedia.org: Cannot log into wikitech - https://phabricator.wikimedia.org/T103939#1427350 (10JanZerebecki) [14:11:06] elee: weird. If I su elee then become xtools, it works [14:12:14] 10Quarry: Quarry shows old username after user was renamed - https://phabricator.wikimedia.org/T73064#1427353 (10He7d3r) I don't know if it is related but, I have a similar problem with the Wikidata Game, which also uses OAuth: https://bitbucket.org/magnusmanske/wikidata-game/issue/153/stats-are-lost-after-a-glo... [15:02:23] valhallasw`cloud: funnily enough, its works now [15:02:28] * elee shrugs [15:02:41] wmflab's entire stack is so... unconventional [15:18:42] 10Quarry: Quarry shows old username after user was renamed - https://phabricator.wikimedia.org/T73064#1427389 (10yuvipanda) It's probably a similar issue - we just cache the name OAuth gives us in the db and do not change it every login. We perhaps should... [18:57:18] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ineuw was modified, changed by Ineuw link https://wikitech.wikimedia.org/w/index.php?diff=169151 edit summary: [18:59:14] * CP678|away is away: This is a manual computer virus. Please copy paste me in your away message. I'm not here right now. [18:59:54] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ineuw was modified, changed by Ineuw link https://wikitech.wikimedia.org/w/index.php?diff=169152 edit summary: [19:02:31] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ineuw was modified, changed by Ineuw link https://wikitech.wikimedia.org/w/index.php?diff=169153 edit summary: [19:04:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ineuw was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=169154 edit summary: [19:20:26] looking for proofread page data from wikisource. tried the text 'table' but it doesn't exist even though it's listed in the schema [19:20:59] ineuw: text is not available in the database [19:21:06] ineuw: you'll have to use the api [19:21:53] which api please? [19:22:07] the mediawiki api [19:22:17] e.g. en.wikisource.org/w/api.php [19:22:24] thanks [20:36:17] 10Tool-Labs-tools-Other: Https bug with Contributions by User to a page - https://phabricator.wikimedia.org/T104812#1428049 (10Krenair) [22:48:09] elee: still having ssh troubles? [22:56:58] YuviPanda: what was that tool that checked instance puppet roles and vars? [22:57:07] ^ that you made [22:58:04] Negative24: tools.wmflabs.org/watroles/role/rolename [22:58:09] Replace filename [22:58:13] Err [22:58:13] Rolename [22:58:43] Thanks [22:58:56] have you published the source? [23:15:28] Negative24: yeah. Github/yuvipanda/watroles [23:22:03] YuviPanda: very nice thanks again [23:22:20] Yw