[00:00:20] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Quietmouse was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1173756 edit summary: [00:50:39] 06Labs, 10Tool-Labs, 06Operations: puppetize legacy toolserver mail aliases - https://phabricator.wikimedia.org/T153510#2882918 (10Dzahn) [00:50:59] 06Labs, 10Tool-Labs, 06Operations: puppetize legacy toolserver mail aliases - https://phabricator.wikimedia.org/T153510#2882933 (10Dzahn) [00:51:10] Attention: Scheduled maintaince for grrrit-wm in 9 minutes !!! [00:57:51] Attention: Scheduled maintenance for mutante in 8 minutes [01:20:39] I need an tools lab admin [01:20:46] ASAP [01:26:07] Zppix: what? [01:28:06] I need kubectl reran for grrrit-wm as i needed to kill it [01:30:33] legoktm: [01:31:32] what did you do? [01:32:20] Zppix: ? [01:32:54] legoktm he did https://wikitech.wikimedia.org/wiki/Grrrit-wm#Building.2FDeploying [01:33:00] kubectl stop replicationcontroller grrrit [01:33:02] no [01:33:05] why was it stopped? [01:33:12] i doint know why he did that [01:33:21] Nickserv configs [01:33:23] i wasent arround at the time he ran that [01:33:45] Zppix: you need to coherently explain what you did [01:34:19] I grouped the test bot nick to the grrrit-wm acct [01:34:25] See the sal for grrrit-wm [01:34:30] I grouped the test bot nick to the grrrit-wm acct [01:35:03] why did you need to stop the bot for that? [01:35:18] So i could login as the bot to do that [01:35:26] ... [01:35:34] you can log into the same account multiple times [01:35:38] you don't need to kill it [01:35:50] I tried and it wouldnt let me [01:35:53] So i killed it [01:36:57] if you don't know what you're doing, stop [01:38:34] ^ wtf [01:39:09] I dont like that [01:40:46] Whats going on... [01:40:48] that's me [01:40:51] I'm trying to fix it [01:40:53] Og [01:40:55] Oh [01:41:18] Phew i was gonna say i didnt change any code [02:01:59] !log tools.lolrrit-wm grrrit-wm is currently running in legoktm's mosh session on tools-login [02:02:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [02:13:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:49:26] Labs tools works slow last 2 days, what's the reason? [03:05:01] (03PS1) 10Yuvipanda: Remove Dockerfile [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327894 [03:05:04] (03PS1) 10Yuvipanda: Add kubernetes deployment yaml file [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327895 [03:05:07] (03PS1) 10Yuvipanda: Remove Stale info from README [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327896 [03:08:54] legoktm: can you also merge ^ [03:09:02] sure [03:09:40] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:18:24] 06Labs, 10Tool-Labs: Lags and non-working mono on Labs - https://phabricator.wikimedia.org/T153516#2883080 (10MaxBioHazard) [03:24:05] 06Labs, 10Tool-Labs: Backup and/or puppetize @toolserver.org mail forwards - https://phabricator.wikimedia.org/T136225#2883098 (10scfc) [03:24:07] 06Labs, 10Tool-Labs, 06Operations: puppetize legacy toolserver mail aliases - https://phabricator.wikimedia.org/T153510#2883100 (10scfc) [03:27:58] is there anything going on at the login servers? [03:28:19] I've been trying to git clone something from gerrit, stuck at "cloning into" for few minutes now. Usually doesn't happen [03:28:36] it also took me multiple tries to `become` a tool [03:31:20] also tried GitHub, cloning at 2KB/s [03:40:35] 06Labs, 10Tool-Labs: RfA Vote Counter: Error - https://phabricator.wikimedia.org/T153517#2883104 (10JustBerry) [04:05:40] 10Tool-Labs-tools-Other: RfA Vote Counter: Error - https://phabricator.wikimedia.org/T153517#2883139 (10scfc) a:03JackPotte [04:12:43] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [04:22:45] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:35:49] (03CR) 10Legoktm: [C: 032] Remove Dockerfile [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327894 (owner: 10Yuvipanda) [04:36:05] (03CR) 10Legoktm: [C: 032] Add kubernetes deployment yaml file [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327895 (owner: 10Yuvipanda) [04:36:16] (03CR) 10Legoktm: [C: 032] Remove Stale info from README [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327896 (owner: 10Yuvipanda) [04:36:19] (03Merged) 10jenkins-bot: Remove Dockerfile [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327894 (owner: 10Yuvipanda) [04:36:37] legoktm: thanks :) [04:36:38] (03Merged) 10jenkins-bot: Add kubernetes deployment yaml file [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327895 (owner: 10Yuvipanda) [04:36:49] mhm [04:36:51] (03Merged) 10jenkins-bot: Remove Stale info from README [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327896 (owner: 10Yuvipanda) [04:39:50] when you are struggling and yuvipanda sends a broadcast... ;) [04:40:20] dargasea: yeah, it's worse than usual so am doing some NFS things [04:44:23] dargasea: is it any better now? [04:45:07] yuvipanda, doesn't seem so; home dir ls took 5+s [04:45:45] same for tab completions [04:45:52] hmm, I see that too [04:48:08] dargasea: how about now [04:49:00] ls is back to .2s now for me [04:49:09] woah, things are back to normal now [04:49:10] !log tools turned on lookupcache again for bastions [04:49:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:49:47] !log tools.gpfsexifbot kill process running on tools-login, was using up all NFS bandwidth [04:49:47] Unknown project "tools.gpfsexifbot" [04:50:21] !log tools.gpfsexif kill process running on tools-login, was using up all NFS bandwidth [04:50:21] Unknown project "tools.gpfsexif" [04:50:37] !log tools.gpsexif kill process running on tools-login, was using up all NFS bandwidth [04:50:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gpsexif/SAL [04:50:43] that should work [04:51:06] that being said, are there any policies in place to prevent people from running stuff on tools? [04:51:22] I do recall noticing gps when I was looking at top [04:51:42] dargasea: yeah, for CPU and RAM limits, but this one was exhausting available NFS IO, for which we can only do per-host limits as of now [04:52:00] a year ago, this single process would've killed NFS in all of labs and made everything slow for everyone - today it only affects this one instance [04:53:07] ah, I see [04:53:13] thanks so much for the fix [04:53:36] dargasea: yw, thanks for bringing it to our attention [06:52:43] 06Labs: New entries in meta_p.wiki are missing a URL - https://phabricator.wikimedia.org/T142759#2883296 (10Krinkle) >>! In T142759#2824239, @jcrespo wrote: >>> db slice > > If you want to be efficient, query all wikis on the same server I'm aware of this implementation detail, but so far believed that aside f... [07:00:09] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:03:03] (03PS1) 10Krinkle: Fix "dns_get_record()" warning when search contains space or parenthesis [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/327906 [07:04:29] (03CR) 10Krinkle: [C: 032] "Verified in labs." [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/327906 (owner: 10Krinkle) [07:05:15] (03Merged) 10jenkins-bot: Fix "dns_get_record()" warning when search contains space or parenthesis [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/327906 (owner: 10Krinkle) [07:09:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:07] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:35] PROBLEM - Free space - all mounts on tools-docker-builder-03 is CRITICAL: CRITICAL: tools.tools-docker-builder-03.diskspace.root.byte_percentfree (<20.00%) [08:05:40] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:00:29] PROBLEM - Puppet run on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:40:29] RECOVERY - Puppet run on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:26:16] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2883607 (10valhallasw) [11:26:19] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Install python-requests-oauthlib on labs - https://phabricator.wikimedia.org/T130529#2883610 (10valhallasw) [11:26:43] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2876092 (10valhallasw) It //is// installed, just not on precise nodes, as ubuntu does not package python-requests-oauthlib for precise. [11:35:58] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2883618 (10Urbanecm) Thanks for the message, forcing trusty hosts works. [11:51:28] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2883628 (10zhuyifei1999) >>! In T153308#2883618, @Urbanecm wrote: > Thanks for the message, forcing trusty hosts works. I thought trusty is the default nowa... [11:52:47] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2883629 (10Urbanecm) It should (I got warning "trusty is default") but jsub python ~/pwb/scripts/login.py don't work, with jsub -l release=trusty python ~/pwb... [12:00:25] 10Tool-Labs-tools-Other: RfA Vote Counter: Error - https://phabricator.wikimedia.org/T153517#2883636 (10JackPotte) This tool should be totally rewritten like or even merged into https://github.com/x-tools/SuperCount. So it won't be finished this year... [12:01:13] 10Tool-Labs-tools-Other: RfA Vote Counter: Error - https://phabricator.wikimedia.org/T153517#2883637 (10JackPotte) p:05Triage>03Low [13:32:15] 10PAWS: PAWS does not enforce HTTPS - https://phabricator.wikimedia.org/T152636#2883759 (10WikidataFacts) [14:10:06] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2883835 (10yuvipanda) I upgraded the terminal component many many versions, so try this one out again? [14:10:40] 10PAWS: PAWS terminal doesn't support alt- on Mac OS X keyboard - https://phabricator.wikimedia.org/T139739#2883836 (10yuvipanda) Try now? I just deployed a brand new version of the terminal component [14:11:05] 10PAWS: Add "less" and "wget" to docker image - https://phabricator.wikimedia.org/T132070#2883838 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Yup! [14:13:42] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2883841 (10Urbanecm) It works using normal mode of Google Chrome, using incognito window (so I don't need to log out and then log in again) it does not work. [14:13:59] 10PAWS: Paste does not work in PAWS terminal - https://phabricator.wikimedia.org/T120633#2883843 (10yuvipanda) 05Open>03Resolved a:03yuvipanda I've deployed a much newer version of xterm.js now, and this should work. Please re-open if it does not! Thank you for your patience [14:14:17] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2883846 (10yuvipanda) (you need to go to control panel, then shut down your server and start it again to see the changes) [14:19:03] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2883849 (10Urbanecm) That was the reason why I didn't see anything. Thanks for the fix! [14:20:00] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2883850 (10yuvipanda) Cool! So it works fine for you in all the configurations you care about, right? [14:20:37] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2883851 (10Urbanecm) Yes! [14:21:14] 10PAWS, 15User-Urbanecm: [] can't be inserted into the PAWS - https://phabricator.wikimedia.org/T153457#2883852 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Ok :) Thanks for reporting! [16:45:47] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2883952 (10bd808) >>! In T153308#2883629, @Urbanecm wrote: > It should (I got warning "trusty is default") but jsub python ~/pwb/scripts/login.py don't work,... [16:52:42] 06Labs: Request creation of twl-staging labs project - https://phabricator.wikimedia.org/T153549#2883959 (10ThatAndromeda) [17:06:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:09:39] 06Labs, 10Tool-Labs, 10Pywikibot-core, 15User-Urbanecm: Install requests_oauthlib at Toollabs exec nodes - https://phabricator.wikimedia.org/T153308#2884004 (10Urbanecm) Strange, this bug doesn't appear right now... [18:58:57] 10Quarry: Add forking hierarchy view for queries - https://phabricator.wikimedia.org/T153553#2884061 (10Base) [19:02:25] 10Quarry: Allow to purge query off its parent - https://phabricator.wikimedia.org/T153554#2884073 (10Base) [19:04:42] 10Quarry, 10Analytics-Wikimetrics: Include Tulu Wikipedia in Metrics and Quarry - https://phabricator.wikimedia.org/T148950#2737572 (10Base) I have zero clue about wikimetrics but quarry seem to support tcywiki_p at the moment, at least ``` use tcywiki_p; show tables; ``` did return the table list. [19:10:45] 10Quarry, 10Analytics-Wikimetrics: Include Tulu Wikipedia in Metrics and Quarry - https://phabricator.wikimedia.org/T148950#2737572 (10Krenair) The Quarry code has no control over this, it relies on the labs DB replicas, which was done in T142223 [19:11:38] (03PS1) 10BryanDavis: Fix account creation bug [labs/striker] - 10https://gerrit.wikimedia.org/r/327943 [19:11:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:13:26] (03PS2) 10BryanDavis: Check and enforce OATH account protection [labs/striker] - 10https://gerrit.wikimedia.org/r/327786 (https://phabricator.wikimedia.org/T144712) [20:07:47] (03CR) 10Andrew Bogott: [C: 032] Fix account creation bug [labs/striker] - 10https://gerrit.wikimedia.org/r/327943 (owner: 10BryanDavis) [20:09:09] (03Merged) 10jenkins-bot: Fix account creation bug [labs/striker] - 10https://gerrit.wikimedia.org/r/327943 (owner: 10BryanDavis) [20:11:29] !log tools.noclaims Set up the bot with a clone of https://github.com/multichill/toollabs and a symlinked pywikibot (git clone is broken see phab:T151351 ) [20:11:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.noclaims/SAL [20:11:33] T151351: Fresh clone of pywikibot from gerrit fails with error: RPC failed; result=56, HTTP code = 200 on Toollabs - https://phabricator.wikimedia.org/T151351 [20:22:13] !log tools.zppixbot restarted web service to clear cache [20:22:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [20:37:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:53:58] Hey yuvipanda, I see you switched one of my projects to Kubernetes. How do I tell it to serve files like *.sql and *.log as plain text instead of offering it for download? [20:55:54] !log tools.noclaims Moved the two jobs here (one in the morning and one in the evening) and updated https://www.wikidata.org/wiki/User:NoclaimsBot [20:55:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.noclaims/SAL [20:57:25] 10PAWS: Add possibility to run other users notebooks by copying to own folder - https://phabricator.wikimedia.org/T139036#2884155 (10Abbe98) You can achieve this using wget and paws-public.wmflabs.org: ``` wget http://paws-public.wmflabs.org/paws-public/44645351/wikibot.py ``` This does not work for files re... [21:00:33] 06Labs, 10Tool-Labs, 10Gerrit, 10Pywikibot-core: Fresh clone of pywikibot from gerrit fails with error: RPC failed; result=56, HTTP code = 200 on Toollabs - https://phabricator.wikimedia.org/T151351#2884156 (10Multichill) >>! In T151351#2815002, @Paladox wrote: > Try this http://stackoverflow.com/questions... [21:01:55] 10PAWS, 07Upstream: PAWS can't edit SQL files - https://phabricator.wikimedia.org/T146920#2884158 (10Abbe98) [21:16:52] 10PAWS, 07Upstream: PAWS can't edit SQL files - https://phabricator.wikimedia.org/T146920#2884164 (10Abbe98) Upstream issue: https://github.com/jupyter/notebook/issues/1106 Aka, this happens for all files your browser does download by dafault. [21:37:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:31] multichill: it should have the exact same behavior as gridengine backed ones [21:46:19] * yuvipanda needs to fall asleep soon [21:58:06] yuvipanda: Hmm, under one account I get .sql files as plain text, the other it's offered for download [21:58:26] I remeber playing around with mime types a long long time ago, but not sure where that's configured these days [21:58:40] multichill: hmm, same .lighttpd.conf for both k8s and gridengine [21:58:48] maybe one tool has one and the otehr does not? [21:59:07] Ah, there it is! [22:00:49] yuvipanda: Any reload/restarting needed to activate it? [22:01:02] multichill: yeah, just 'webserivce restart' should do [22:01:05] *webservice restart [22:01:59] Excellent. Looked over the .lighttpd.conf [22:02:05] Now it works [22:02:09] \o/ cool [22:02:18] multichill: any other candidates for moving over to the k8s backend? :) [22:03:08] somebody only to restart grrrit serverside? [22:03:22] the connection is very unstable after the last restart from IRC [22:03:23] !log tools.noclaims Added .lighttpd.conf and webservice restart so that logs are now send as "text/plain;charset=UTF-8" [22:03:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.noclaims/SAL [22:03:38] while other things connected from the host are stable, so I guess the code has a problem [22:03:50] Does it do simple php with database connection yuvipanda? [22:05:03] multichill: yup [22:05:26] Anyway, enough for today. Moved everything I wanted to move to https://www.wikidata.org/wiki/User:NoclaimsBot [22:06:01] \o/ ok! :) [22:06:21] I'll go to sleep then [22:06:21] night [22:06:51] Thanks for the help. I'll have a look at Kubernetes some other day [22:07:02] Looks good! And I should probably set up the log rotation too [22:07:45] yuvipanda: You're home for the holidays? [22:18:35] RECOVERY - Free space - all mounts on tools-docker-builder-03 is OK: OK: All targets OK [22:20:22] !log tools.lolrrit-wm restarting pod, seems to be having ping handling issues? [22:20:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL [22:21:24] sorry, this was me, not a connection problem [22:22:19] (03CR) 10Alex Monk: "bump random commit to test bot" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/283944 (owner: 10Luke081515) [22:22:55] huh, is that one old [22:22:59] yes [22:23:24] 200k commit id [22:24:05] Krenair: that reconnect before was caused due T148789 I think. [23:38:42] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:48:46] (03Draft1) 10Paladox: Remove grrrit-wm: force-restart [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/328025 [23:48:49] (03Draft2) 10Paladox: Remove grrrit-wm: force-restart [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/328025