[00:33:28] <doctaxon>	 are there issues with tools-login.wmflabs.org ssh ?
[00:34:16] <doctaxon>	 it lags saving files in nano
[00:34:18] <doctaxon>	 and so on
[00:35:07] <doctaxon>	 Coren - anything on?
[00:36:11] <Krenair>	 doctaxon, yes, try now
[00:36:32] <doctaxon>	 yes, it's interrupting meanwhile
[00:37:26] <Krenair>	 although.... I know ldap broke which caused issues logging in, but I don't think that'd cause file writing to be effected
[00:38:11] <doctaxon>	 Krenair: last two days it emptied the file to zero !!!
[00:40:09] <doctaxon>	 Krenair: do you try ldap tests these days?
[00:40:34] <Krenair>	 I can log in to labs at the moment
[00:40:46] <doctaxon>	 yes, now it's okay
[00:40:57] <doctaxon>	 but it interrupts then and then
[00:42:44] <Krenair>	 it  what?
[00:44:27] <doctaxon>	 it is lagging time to time
[00:44:44] <andrewbogott>	 doctaxon: ‘time to time’ meaning a few times per day or a few times per hour?
[00:45:50] <doctaxon>	 I noticed it three times this hour
[00:46:20] <andrewbogott>	 ok — please let me know if it happens again soon?
[00:46:34] <doctaxon>	 sure
[00:46:50] <doctaxon>	 can you tell me anything about that?
[00:47:33] <andrewbogott>	 We’ve had issues with labs seizing up over the last few days
[00:47:50] <andrewbogott>	 it’s not yet well-understood.  But one possible culprit is the ldap server which was just now restarted.
[00:47:52] <doctaxon>	 I noticed
[00:48:16] <andrewbogott>	 So, maybe it’s cured at least for another 12 hours or so?
[00:48:22] <andrewbogott>	 If it continues to stutter, that’s a datapoint.
[00:49:02] <doctaxon>	 I got a ldap error with "become"
[00:49:44] <doctaxon>	 but I use tools-login.wmflabs.org
[00:50:27] <doctaxon>	 "cannot connect"
[00:50:53] <doctaxon>	 about 00:30 UTC
[01:13:43] <doctaxon>	 andrewbogott: it lags right now
[01:13:53] <doctaxon>	 saving a file with nano
[01:14:18] <andrewbogott>	 doctaxon: ok — I see an alert in another channel which is surely related.  Thanks!
[01:14:26] <doctaxon>	 oh
[01:16:25] <andrewbogott>	 still stuck?  Or just slow?
[01:16:27] <doctaxon>	 stuck
[01:16:28] <YuviPanda>	 yeah I think NFS is stuck
[01:17:02] <doctaxon>	 now the file is written
[01:17:59] <chasemp>	 I am on my phone are you guys around to deal with tools home?
[01:18:21] <YuviPanda>	 chasemp: yeah, just LDAP failing.
[01:18:25] <YuviPanda>	 chasemp: I'll call you if need be.
[01:18:26] <chasemp>	 duunno what it would be thia time
[01:18:27] <chasemp>	 ah
[01:18:54] <chasemp>	 any clue what happened to ldap?
[01:19:38] <YuviPanda>	 chasemp: it fell over (FD exhaustion)
[01:19:42] <YuviPanda>	 got restarted
[01:20:25] <chasemp>	 Oh gotcha, how do we know fd issues?  I will note similar time of night to last time as well
[01:20:38] <chasemp>	 i wonder if some job hammers it
[01:21:10] <chasemp>	 fwiw if we get desperate an idletimeout did work to cull bad clients
[01:21:52] <chasemp>	   Granted 10s was aggressive, maybe 300s?  Anyway i am a call away if i can help
[01:22:14] <YuviPanda>	 chasemp: yes, there's a 10s idletimeout now.
[01:22:23] <YuviPanda>	 chasemp: yup, parav.oid is on it now.
[01:22:33] <paravoid>	 10 *minutes*
[01:22:45] <YuviPanda>	 oh
[01:22:47] <YuviPanda>	 ok
[01:25:32] <chasemp>	 Ok makes sense i hope that does it, thanks paravoid
[01:25:33] <paravoid>	 as I said in the other channel... https://graphite.wikimedia.org/render/?width=586&height=308&target=servers.seaborgium.openldap.conns.current
[01:25:33] <paravoid>	 it lags, because... our pipeline sucks
[01:25:34] <paravoid>	 but better than nothing
[01:25:34] <paravoid>	 I'll let it collect some data and then we can create a grafana dashboard
[01:29:08] <doctaxon>	 API does not work?
[01:33:32] <paravoid>	 chasemp: you configured diamond initially, right?
[01:33:39] <paravoid>	 why are we pushing to statsd and not to graphite directly?
[01:35:19] <chasemp>	 statistics i imagine at this point but that is all ori and godog at this point
[01:36:33] <chasemp>	 95th percentile etc, i think ppl prefer it mainly and also there is some statsdlb and statsite layering i dont entirely get now
[01:36:42] <Negative24>	 igloo seems to be stuck on tools. Known?
[01:37:03] <chasemp>	 diffusion of load?  Not sure I didnt have much to do with it
[01:37:04] <YuviPanda>	 doctaxon: what do you mean by 'API does not work'?
[01:37:10] <YuviPanda>	 Negative24: 'igloo'?
[01:37:21] <doctaxon>	 oh it was my fault sorry
[01:37:29] <YuviPanda>	 doctaxon: np :)
[01:37:55] <chasemp>	 Ok batteries at 3% i gotta get to charger 
[01:39:44] <Negative24>	 YuviPanda: [[WP:Igloo]] and same name on tools
[01:40:23] <YuviPanda>	 Negative24: http://tools.wmflabs.org/igloo/ wfm?
[01:41:27] <Negative24>	 YuviPanda: that's the web server. It's a gadget on enwiki and isn't responding to requests
[01:41:40] <Negative24>	 I don't know if that's a tools problem or igloo
[01:41:58] <wikibugs>	 6Labs, 10netops, 6operations, 5Patch-For-Review: Create labs baremetal subnet? - https://phabricator.wikimedia.org/T121237#1874862 (10Dzahn)
[01:42:04] <YuviPanda>	 I can see if there's a problem on the backend if you can point me to a specific request that's failing
[01:42:09] <YuviPanda>	 I see that the webservice is up and working
[01:44:32] <Negative24>	 YuviPanda: nm, its probably not something with tools. All I know is that chrome is waiting on tools.wikimedia.org forever
[01:44:33] <wikibugs>	 6Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 7Mobile: Install MobileFrontend on wikitech - https://phabricator.wikimedia.org/T87633#1874870 (10Dzahn) I also think that's what we have wikitech-static for.  But if we decline this, then fine, then we should just delete the DNS entry,
[01:44:49] <YuviPanda>	 Negative24: hit refresh and see what happens :)
[01:45:24] <Negative24>	 still going
[04:56:28] <wikibugs>	 6Labs, 10Tool-Labs, 10Gerrit: git clone operations/mediawiki-config on tool labs fail: recursion detected in die_errno handler - https://phabricator.wikimedia.org/T106393#1875009 (10liangent) I can now reproduce it in VisualEditor:  ``` tools.liangent-php@tools-bastion-01:~/mw/extensions$ cat clone.sh #!/bin...
[05:11:49] <Glaisher>	 ssh to bastion.wmflabs.org seems to be failing
[05:12:11] <Glaisher>	 Connection closed by 208.80.155.129
[05:13:34] <Glaisher>	 tools-login is working fine
[05:15:03] <Krenair>	 Coren, ^
[05:15:12] <Krenair>	 YuviPanda, ^
[05:18:21] <Glaisher>	 https://phabricator.wikimedia.org/P2417
[05:35:09] <Krenair>	 Glaisher, I sent Coren a message
[05:38:25] <Glaisher>	 Krenair: Thanks.
[05:38:29] <Glaisher>	 I just tried sshing on my windows pc as well but it's failing there as well
[05:39:22] <Krenair>	 yeah it doesn't work for me either
[05:39:30] <Krenair>	 definitely something up on the server side
[05:41:03] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Create a larger newsletter-test instance in labs - https://phabricator.wikimedia.org/T120516#1875027 (10Glaisher) Thank you very much bd808! I just set up the web proxy at https://newsletter-test.wmflabs.org again. But due to some issues with ssh, I haven't been able t...
[05:41:36] <Glaisher>	 should probably file a bug about it then
[05:43:28] <wikibugs>	 6Labs: sshing to bastion.wmflabs.org fails with "Connection closed by 208.80.155.129" - https://phabricator.wikimedia.org/T121302#1875028 (10Glaisher) 3NEW
[06:48:43] <ebernhardson>	 same, just noticed i can't login :(
[06:50:37] <wikibugs>	 6Labs: sshing to bastion.wmflabs.org fails with "Connection closed by 208.80.155.129" - https://phabricator.wikimedia.org/T121302#1875051 (10EBernhardson) Doesn't seem isolated to Glaisher, i just noticed same problem
[07:09:52] <wikibugs>	 6Labs, 10Datasets-Archiving, 10Datasets-General-or-Unknown, 10Labs-Infrastructure, 10Wikidata: [Bug] Wikidata JSON dumps gets deleted after every new Wikidata dump - https://phabricator.wikimedia.org/T107226#1875055 (10Hydriz) 5Invalid>3Open Clearly it hasn't been fixed.
[07:10:01] <wikibugs>	 6Labs, 10Datasets-Archiving, 10Datasets-General-or-Unknown, 10Labs-Infrastructure, 10Wikidata: [Bug] Wikidata JSON dumps gets deleted after every new Wikidata dump - https://phabricator.wikimedia.org/T107226#1875058 (10Hydriz) a:5Hydriz>3None
[07:34:39] <shinken-wm>	 PROBLEM - Host tools-worker-04 is DOWN: CRITICAL - Host Unreachable (10.68.16.122)
[09:22:13] <YuviPanda>	 Krenair: Glaisher around? can you try sshing now?
[09:23:48] <zhuyifei1999_>	 YuviPanda: works for me now. It didn't a few hours ago
[09:23:58] <YuviPanda>	 zhuyifei1999_: where are you sshing to?
[09:24:04] <YuviPanda>	 I just hand-hacked a fix in
[09:24:27] <zhuyifei1999_>	 tools-bastion-01.eqiad.wmflabs via bastion
[09:25:31] <YuviPanda>	 ah ok
[09:26:21] <zhuyifei1999_>	 tools-dev.wmflabs.org worked when tools-bastion-01.eqiad.wmflabs via bastion failed (ik they are different hosts, but I didn't check tools-login.wmflabs.org or  tools-bastion-02.eqiad.wmflabs)
[09:26:39] <YuviPanda>	 yeah, .wmflabs.org go directly
[09:26:46] <zhuyifei1999_>	 ik
[09:26:57] <zhuyifei1999_>	 (just to test the bug)
[09:26:58] <YuviPanda>	 I know that tools-login.wmflabs.org worked for me
[09:29:25] <zhuyifei1999_>	 hmm why isn't there a wikibugs message for your comment to T121302?
[09:30:08] <YuviPanda>	 ugh
[09:30:10] <YuviPanda>	 it probably died too
[09:30:29] <zhuyifei1999_>	 ctcp the bot works
[09:30:49] <YuviPanda>	 I restarted both the bots
[09:32:36] <zhuyifei1999_>	 also, could you check if the crontab on tools-submit is working? One of my jobs that's supposed to submit 3 times an hour had no log output nor qstat output since 3:10 UTC
[09:33:17] <zhuyifei1999_>	 lst submit should be 3 minutes ago, still nothing
[09:33:25] <zhuyifei1999_>	 manual jsub works
[09:33:28] <zhuyifei1999_>	 *last
[09:34:25] <YuviPanda>	 ugh
[09:34:29] <YuviPanda>	 what's your tool name
[09:34:38] <zhuyifei1999_>	 yifeibot
[09:34:45] <zhuyifei1999_>	 the job is flr
[09:35:13] <zhuyifei1999_>	 for this bot https://commons.wikimedia.org/wiki/Special:Contributions/FlickreviewR_2
[09:38:40] <shinken-wm>	 PROBLEM - Puppet failure on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[09:41:30] <YuviPanda>	 ok, I'm not sure at all why cron isn't working
[09:41:38] <zhuyifei1999_>	 :/
[09:42:40] <YuviPanda>	 oh
[09:42:42] <YuviPanda>	 Dec 12 09:42:02 tools-submit CRON[24428]: Permission denied
[09:42:48] <YuviPanda>	 Dec 12 09:42:10 tools-submit nslcd[29853]: [e85ae0] <passwd(all)> passwd entry uid=80686,ou=people,dc=wikimedia,dc=org denied by validnames option: "80686"
[09:43:02] <YuviPanda>	 zhuyifei1999_: can you file a bug?
[09:43:21] <zhuyifei1999_>	 k
[09:46:31] <wikibugs>	 6Labs, 10Tool-Labs: cron fail on tools-submit - https://phabricator.wikimedia.org/T121305#1875157 (10zhuyifei1999) 3NEW
[09:47:43] <wikibugs>	 6Labs, 10Tool-Labs: cron fail on tools-submit - https://phabricator.wikimedia.org/T121305#1875165 (10yuvipanda) p:5Triage>3High Seems LDAP related:  ```Dec 12 09:42:02 tools-submit CRON[24437]: Permission denied Dec 12 09:42:02 tools-submit CRON[24428]: Permission denied Dec 12 09:42:02 tools-submit CRON[2...
[09:53:16] <YuviPanda>	 zhuyifei1999_: can you see if any of your crons have run?
[09:53:33] <zhuyifei1999_>	 like right now?
[09:54:06] <YuviPanda>	 zhuyifei1999_: in the last few hours
[09:54:13] <zhuyifei1999_>	 hmm
[09:56:06] <zhuyifei1999_>	 most jobs don't produce very obvious log outputs
[09:58:22] <zhuyifei1999_>	 nope, one job that's 17 * * * * has no "job already running messages" since [Sat Dec 12 02:17:03 2015]
[09:59:04] <YuviPanda>	 ok
[09:59:05] <zhuyifei1999_>	 it's continuous, so it should have that message every hour
[10:03:01] <Merlissimo>	 YuviPanda: all my cron sge submittions are not working, too
[10:03:55] <wikibugs>	 6Labs: sshing to bastion.wmflabs.org fails with "Connection closed by 208.80.155.129" - https://phabricator.wikimedia.org/T121302#1875203 (10fgiunchedi) "fixed" also bastion-restricted-01:  ``` root@bastion-restricted-01:~# cat /etc/security/access.conf -:ALL EXCEPT (ops) root:ALL root@bastion-restricted-01:~# p...
[10:04:23] <YuviPanda>	 I'm relocating, I'll take a look in about 30mins again. sorry.
[10:04:35] <Merlissimo>	 last run was at at 2:51 UTC
[10:06:03] <zhuyifei1999_>	 k
[10:06:11] <Merlissimo>	 just for your info, no emergency request for me
[10:06:21] <YuviPanda>	 scratch that, I'm back and looking
[10:06:26] <YuviPanda>	 thanks Merlissimo, last run info is helpful
[10:06:42] <YuviPanda>	 this breaks a lot of bots wikis rely on doesn't it
[10:07:46] <zhuyifei1999_>	 lol unless it's a continuous bot
[10:08:30] <YuviPanda>	 yeah
[10:08:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-docker-builder-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:08:45] <YuviPanda>	 !log tools restarted cron on tools-submit
[10:08:49] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[10:09:43] <zhuyifei1999_>	 hmm next crontab submit is in a minute
[10:10:00] <YuviPanda>	 let me see if the restart helped
[10:10:51] <zhuyifei1999_>	 ... and qstat says nothing about flr
[10:11:49] <wikibugs>	 6Labs, 10Tool-Labs: cron fail on tools-submit - https://phabricator.wikimedia.org/T121305#1875209 (10Stigmj) Same problem for my account. Last task run was 03:10.
[10:14:19] <YuviPanda>	 godog: I think the cron issues are related to PAM
[10:15:37] <godog>	 YuviPanda: could be
[10:16:55] <YuviPanda>	 zhuyifei1999_: have a bot running anytime soon?
[10:17:44] <zhuyifei1999_>	 I can rescedule
[10:18:01] <YuviPanda>	 zhuyifei1999_: <3 that'd be lovely
[10:18:39] <zhuyifei1999_>	 actually I'll do a simple bash script
[10:21:19] <zhuyifei1999_>	 job "jsubtest" for every five minutes
[10:23:52] <zhuyifei1999_>	 so next run is in two minutes, outputing date & time to jsubtest.err
[10:24:14] <YuviPanda>	 I see other bots running now
[10:25:43] <zhuyifei1999_>	 yep got one Sa 12. Dez 10:25:05 UTC 2015
[10:26:07] <YuviPanda>	 \o/
[10:26:09] <YuviPanda>	 cool
[10:26:11] <YuviPanda>	 let me update bug
[10:26:42] <wikibugs>	 6Labs, 10Tool-Labs: cron fail on tools-submit - https://phabricator.wikimedia.org/T121305#1875213 (10yuvipanda) https://gerrit.wikimedia.org/r/#/c/258663/ was hand-reverted on tools-submit, bringing cron back. Will need more thorough investigation when I'm more awake...
[10:28:56] <zhuyifei1999_>	 "I'm more awake" lol
[10:29:41] <YuviPanda>	 hehe, it's 2:30 AM
[10:29:47] <YuviPanda>	 and this week has had me sleep not very much
[10:31:11] <zhuyifei1999_>	 :/
[10:31:50] <zhuyifei1999_>	 2:30 AM sound like a very bad time to stay awake
[10:32:43] <YuviPanda>	 part of being ops, I guess :) it always falls apart when you go 'ok I will take one look before I go to sleep'
[10:32:52] <YuviPanda>	 then bam bam bam everything is on fire and you gotta put it out
[10:33:32] <zhuyifei1999_>	 lol
[10:34:16] <zhuyifei1999_>	 now I wonder how many hours jynus sleep every day
[10:34:32] <jynus>	 lol
[10:35:08] <YuviPanda>	 jynus never sleeps
[10:35:13] <jynus>	 what is this sleeping that you continue talking about
[10:35:31] <YuviPanda>	 'jynus never sleeps' I ask google
[10:35:42] <YuviPanda>	 'Genius never sleeps' it tells me: https://www.psychologytoday.com/blog/mr-personality/201107/creative-insomnia-genius-never-sleeps
[10:36:01] <jynus>	 you are definitly a genious, I am not
[10:37:02] <YuviPanda>	 clearly explains why I've no idea what I'm doing :D
[10:42:23] <zhuyifei1999_>	 hmm I just googled that and found "mysql> SELECT *, sleep(10) FROM test ORDER BY rand();" in his blog I think
[10:43:45] <YuviPanda>	 hahah :D
[10:44:48] <jynus>	 I wondered what was the context of that ugly query
[10:45:16] <jynus>	 and it was to show the usage of temporary tables, not for anything productive
[10:46:02] <zhuyifei1999_>	 whatever
[10:46:51] <YuviPanda>	 :D
[10:47:08] <YuviPanda>	 I wonder if there are random sleep() calls in our NFS setup somewhere
[10:49:35] <YuviPanda>	 zhuyifei1999_: btw, PAWS is a lot more stable now
[10:49:40] <YuviPanda>	 I'm going to publicly announce it next week
[10:49:49] <zhuyifei1999_>	 ok let me check
[10:53:49] <zhuyifei1999_>	 hmm the old "test.py" script running forever
[10:54:25] <YuviPanda>	 ?
[10:54:48] <zhuyifei1999_>	 import pywikibot
[10:54:49] <zhuyifei1999_>	 s = pywikibot.Site()
[10:54:49] <zhuyifei1999_>	 p = pywikibot.Page(s, "Test")
[10:54:49] <zhuyifei1999_>	 p.get()
[10:55:18] <zhuyifei1999_>	 same for curl www.google.com
[10:55:21] <YuviPanda>	 zhuyifei1999_: ...
[10:55:23] <YuviPanda>	 wat
[10:55:25] <YuviPanda>	 I thought I had fixed this
[10:55:27] <YuviPanda>	 and I had...
[10:55:46] <zhuyifei1999_>	 maybe I should restart or something?
[10:55:54] <YuviPanda>	 goddamit
[10:55:56] <YuviPanda>	 it's back
[10:56:21] <YuviPanda>	 zhuyifei1999_: I killed yours.
[10:56:28] <YuviPanda>	 I think one node is pretty badly fucked up
[10:56:42] * YuviPanda checks
[10:56:49] <YuviPanda>	 zhuyifei1999_: if you try again you should hopefully get a good node
[10:57:16] <zhuyifei1999_>	 ok
[10:59:30] <YuviPanda>	 zhuyifei1999_: working now?
[11:00:03] <zhuyifei1999_>	 firefox says "The page isn't redirecting properly" on https://tools.wmflabs.org/paws/hub/user/Zhuyifei1999/
[11:00:08] <zhuyifei1999_>	 let me try again
[11:00:37] <zhuyifei1999_>	 Ah I see
[11:00:48] <zhuyifei1999_>	 cached 302 loop
[11:00:51] <YuviPanda>	 ah right
[11:01:03] <YuviPanda>	 did a hard refresh clean them up?
[11:01:14] <zhuyifei1999_>	 https://tools.wmflabs.org/paws/user/Zhuyifei1999/ => https://tools.wmflabs.org/paws/hub/user/Zhuyifei1999/ and back again
[11:01:37] <YuviPanda>	 is it still in that loop?
[11:01:43] <zhuyifei1999_>	 yeah
[11:02:29] <YuviPanda>	 zhuyifei1999_: I just killed everything. try again?
[11:02:48] <zhuyifei1999_>	 ctrl-shift-r doesn't seem to work
[11:02:53] <zhuyifei1999_>	 yep now works
[11:04:26] <YuviPanda>	 I need to make this a lot more robust
[11:04:28] <YuviPanda>	 tomorrow, I guess
[11:04:47] <YuviPanda>	 zhuyifei1999_: do you have working network access now
[11:05:12] <zhuyifei1999_>	 yep now the networks works
[11:05:28] <YuviPanda>	 cool
[11:05:33] <zhuyifei1999_>	 just tested curl www.google.com and python test.py
[11:05:55] <YuviPanda>	 zhuyifei1999_: if you do 'pwb.py login' it'll log you in without asking for your password
[11:06:07] <zhuyifei1999_>	 I didn't notice google has this many scripts lol
[11:06:43] <zhuyifei1999_>	 nice
[11:07:45] <YuviPanda>	 there's even a links browser there now :)
[11:07:52] <zhuyifei1999_>	 appearance logging into other wikis doesn't work
[11:08:06] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/tVkNwnK4/
[11:09:13] <YuviPanda>	 zhuyifei1999_: hmm, I wonder if that's a pywikibot bug
[11:09:28] <YuviPanda>	 zhuyifei1999_: can you try creating a 'user-config.py' file in your homedir and setting myfamily and mylang?
[11:09:34] <YuviPanda>	 and then trying pwb.py login?
[11:09:38] <zhuyifei1999_>	 ok
[11:10:15] <zhuyifei1999_>	 'usernames[family]['*'] = os.environ['JPY_USER']' I wonder why is this limited to only the set family
[11:10:26] <YuviPanda>	 zhuyifei1999_: because ['*'] doesn't work
[11:10:45] <YuviPanda>	 zhuyifei1999_: if there aren't that many families I can list them all tho
[11:11:07] <YuviPanda>	 https://github.com/yuvipanda/paws/blob/master/singleuser/user-config.py is where the source is
[11:11:48] <zhuyifei1999_>	 ik
[11:12:05] <zhuyifei1999_>	 yep with customized user-config.py it works
[11:12:18] <YuviPanda>	 right
[11:12:48] <YuviPanda>	 https://phabricator.wikimedia.org/T120334 is the bug in pwb
[11:15:33] <zhuyifei1999_>	 yeah I see why, https://github.com/wikimedia/pywikibot-core/blob/master/pywikibot/site.py#L1994
[11:16:04] <zhuyifei1999_>	 it's checking if the username in user-config.py and the oauth username match
[11:16:18] <zhuyifei1999_>	 and the former is a None
[11:16:25] <YuviPanda>	 right
[11:16:36] <YuviPanda>	 I wonder if we can just list all the families and set usernames for them
[11:17:06] <YuviPanda>	 https://gerrit.wikimedia.org/r/#/c/257165/ might or might not solve this too I guess
[11:17:39] <zhuyifei1999_>	 I'm not entirely sure about setting on all families. you don't want to tokens to go to third-party wikis
[11:18:32] <YuviPanda>	 all wikimedia families I guess
[11:18:34] <zhuyifei1999_>	 ideally there should be some kind of wmf-wiki selector
[11:19:33] <YuviPanda>	 yeah
[11:21:14] <zhuyifei1999_>	 maybe it's not that hard to implement. If I remember correctly, there's a superclass for all wmf wikis
[11:21:44] <YuviPanda>	 it would be great if you can metnion that in the bug or just send me a Pull request :D
[11:22:41] <zhuyifei1999_>	 ugh, it's not yet implemented in pywikibot-core
[11:28:44] <zhuyifei1999_>	 hmm the operation of loading all family files then check whether it's a subclass WikimediaFamily might be very expensive
[11:28:57] <YuviPanda>	 zhuyifei1999_: can't we just hardcode the list of wikimedia families?
[11:29:06] <YuviPanda>	 I don't think there are gonna be any new families for a while
[11:29:26] <zhuyifei1999_>	 ugh
[11:32:59] <zhuyifei1999_>	 https://github.com/wikimedia/pywikibot-core/search?utf8=%E2%9C%93&q=WikimediaFamily <= everything's here
[11:35:37] <YuviPanda>	 so if we hardcode all the projects specified there...
[11:36:19] <YuviPanda>	 shit, I should sleep
[11:36:26] <YuviPanda>	 I try to make sure I"m asleep by 4AM...
[11:36:30] <YuviPanda>	 might miss that today
[11:36:35] <YuviPanda>	 zhuyifei1999_: do play around and file bugs! <3
[11:36:38] <YuviPanda>	 thank you very much :D
[11:36:55] <zhuyifei1999_>	 ok
[11:36:57] <zhuyifei1999_>	 np
[12:51:31] <wikibugs>	 6Labs: sshing to bastion.wmflabs.org fails with "Connection closed by 208.80.155.129" - https://phabricator.wikimedia.org/T121302#1875352 (10coren) 5Open>3Resolved a:3coren The puppet variable restricted_to was used inconsistently between projects; it was set to the group name everywhere but in project bas...
[13:05:04] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Internal error when creating new user in newsletter-test.wmflabs.org - https://phabricator.wikimedia.org/T119945#1875359 (10Glaisher)
[13:05:07] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Create a larger newsletter-test instance in labs - https://phabricator.wikimedia.org/T120516#1875357 (10Glaisher) 5Open>3Resolved Newsletter extension has also now been enabled. Thanks again.
[13:05:27] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Internal error when creating new user in newsletter-test.wmflabs.org - https://phabricator.wikimedia.org/T119945#1875361 (10Glaisher) 5Open>3Resolved See {T120516}
[13:05:31] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Internal error when creating new user in newsletter-test.wmflabs.org - https://phabricator.wikimedia.org/T119945#1875371 (10Glaisher)
[14:39:22] <wikibugs>	 6Labs: Can't ssh to social-tools1 from bastion-01 - https://phabricator.wikimedia.org/T121313#1875463 (10ashley) 3NEW
[15:28:39] <wikibugs>	 6Labs, 10Tool-Labs: Labs: Move tools-shadow off the same host as tool-master - https://phabricator.wikimedia.org/T103390#1875517 (10Aklapper) >>! In T103390#1808581, @coren wrote: > @yuvipanda: It looks like the test is broken, rather than host distribution.  Does somebody plan to fix the test?
[15:48:44] <grrrit-wm>	 (03CR) 10Aklapper: "Amire80, Ricordisamoa: Could these 13 lines get a review / decision what do here? No updates for 5 months and this patch is rotting..." [labs/tools/translatemplate] - 10https://gerrit.wikimedia.org/r/225267 (owner: 10Ricordisamoa)
[17:29:09] <Urbanecm>	 Hi, I am trying to process this SQL on labs 'select * from categorylinks where cl_to like "Vesnice%" or cl_to like "Město%" limit 10' but output is served in bad way. See screenshot on http://urbanecm.8u.cz/wikipedia/labsSql.png . Can you help me?
[17:29:59] <Reedy>	 why is it a bad way?
[17:31:53] <Urbanecm>	 because I cannot see it. 10 rows must fit in one screen and I can see only the bottom. 
[17:32:54] <greg-g>	 scroll up? :)
[17:34:53] <Urbanecm>	 I think that this isn
[17:34:57] <Urbanecm>	 sorry
[17:35:17] <Urbanecm>	 I think that this isn't normal (see http://urbanecm.8u.cz/wikipedia/labsSql2.png )
[17:40:51] <Urbanecm>	 and, by the way, does somebody know why almost all queries in quarry are in queued state? 
[18:53:12] <Stigmj>	 quarry.wmflabs.org gives 502 Bad gateway when trying to login. Known issue?
[19:10:31] <Luke081515>	 Stigmj: File a phabricator task?
[19:43:11] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/HakanIST was created, changed by HakanIST link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/HakanIST edit summary: Created page with "{{Tools Access Request |Justification=modifying existing tools to fix some caveats (especially rech tool ) |Completed=false |User Name=HakanIST }}"
[19:43:11] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/HakanIST was created, changed by HakanIST link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/HakanIST edit summary: Created page with "{{Tools Access Request |Justification=modifying existing tools to fix some caveats (especially rech tool ) |Completed=false |User Name=HakanIST }}"
[20:44:46] <grrrit-wm>	 (03CR) 10Ricordisamoa: "I find this solution dirty because of manual encode/decodes, and a bit pointless since Tool Labs seem to support Python 2 only. I guess on" [labs/tools/translatemplate] - 10https://gerrit.wikimedia.org/r/225267 (owner: 10Ricordisamoa)
[22:59:45] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/HakanIST was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=226733 edit summary: 
[22:59:45] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/HakanIST was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=226733 edit summary: