[00:42:51] <wikibugs>	 6Labs, 10Tool-Labs, 7Regression: [Regression] BigBrother isn't handling jstart - https://phabricator.wikimedia.org/T105223#1439902 (10scfc)
[00:44:20] <wikibugs>	 6Labs, 10Tool-Labs, 7Regression: [Regression] BigBrother isn't handling jstart - https://phabricator.wikimedia.org/T105223#1438898 (10scfc) I'm pretty certain that during the NFS outage, `bigbrother` was restarted and didn't find an existing job for `ecmabot`, thus running into T88122.
[00:49:52] <Cyberpower678>	 legoktm, ping
[00:50:43] <legoktm>	 Cyberpower678: pong
[00:55:22] <Cyberpower678>	 legoktm, quick question.  Suppose I want to start another PHP script on the exec grid from a PHP script on the exec grid.  Do I use php script.php or do I need jsub as well?
[00:56:22] <legoktm>	 Cyberpower678: last I checked things running from the grid cannot submit more jobs
[00:56:57] <Cyberpower678>	 Really?
[00:56:59] <Cyberpower678>	 Are you sure?
[00:57:23] <legoktm>	 TIAS? ;)
[00:57:47] <Cyberpower678>	 If they're already running on the grid, wouldn't a simple php command run another PHP script on the same node?
[00:57:56] <legoktm>	 ohoh
[00:58:06] <legoktm>	 you want to run a php script from a php script?
[00:58:16] <Cyberpower678>	 Yes.
[00:58:17] <legoktm>	 a) yes that should work
[00:58:28] <legoktm>	 b) why? just require the file and call the same functions?
[00:58:49] <Cyberpower678>	 I want to create subworkers
[00:59:10] <legoktm>	 ...in PHP? :/
[00:59:18] <Cyberpower678>	 Yep.  It's doable. :p
[01:00:24] <legoktm>	 https://secure.php.net/manual/en/class.worker.php
[01:01:57] <Cyberpower678>	 legoktm, thanks. :-)
[01:03:04] <legoktm>	 Cyberpower678: actually, http://symfony.com/doc/current/components/process.html is probably a better idea.
[01:03:28] <legoktm>	 as it turns out, the `pthreads` PHP extension is not actually pthreads.
[01:03:46] <legoktm>	 bbl
[01:57:49] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1439944 (10scfc) @yuvipanda probably meant https://github.com/yuvipanda/watroles.
[04:59:59] <wikibugs>	 6Labs, 7Mobile: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1440007 (10bearND) Hey @yuvipanda. I've got the debug.keystore from you, and built the apk manually.   My last question from my previous comment still remains: How do we switch over to the new machine?...
[08:53:29] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Reenable resolv.conf ndots:2 option on CI instances - https://phabricator.wikimedia.org/T105297#1440502 (10hashar) 3NEW a:3coren
[08:54:05] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1106873 (10hashar) We still have that the Gerrit patch applied on the `integration` labs project. F...
[08:57:13] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1440518 (10hashar) >>! In T92351#1439472, @scfc wrote: > Too bad for the readers Google will bring...
[10:12:43] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Graham87 was created, changed by Graham87 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Graham87 edit summary: Created page with "{{Tools Access Request |Justification=I plan to use it for database queries that would take too long to be done on Quarry, for the moment. I may find other uses for it later...."
[10:19:56] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Graham87 was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=169716 edit summary: 
[10:29:35] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Andrew-WMDE was created, changed by Andrew-WMDE link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Andrew-WMDE edit summary: Created page with "{{Tools Access Request |Justification=WikiLovesDownloads tool for Wikimedia Deutschland. |Completed=false |User Name=Andrew-WMDE }}"
[11:27:00] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Andrew-WMDE was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=169727 edit summary: 
[12:01:45] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Build a simple tool to query which instances have which roles / puppet variables - https://phabricator.wikimedia.org/T103995#1440875 (10yuvipanda) Yes that's what I meant. sorry!
[12:03:44] <Graham87>	 Hi all ... could someone copy the replica.my.cnf file into my tools.graham87 directory? Thanks.
[12:07:11] <sitic>	 Graham87: the replica.my.cnf creation is unfortunately currently broken: https://phabricator.wikimedia.org/T104453
[12:08:48] <Graham87>	 Yeah I noticed that ... should I just wait for it to be fixed?
[12:30:49] <wikibugs>	 10Tool-Labs-tools-Other: usersearch on Tool-Labs lists invalid URLs for Contributions by User to a page - https://phabricator.wikimedia.org/T104812#1440988 (10PrimeHunter) This was also reported at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#User_Contribution_Search  https://tools.wmflab...
[12:35:04] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Sdia45 was created, changed by Sdia45 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Sdia45 edit summary: Created page with "{{Tools Access Request |Justification=To create math extensions that work with mathoid. |Completed=false |User Name=Sdia45 }}"
[12:40:09] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Sdia45 was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=169735 edit summary: 
[13:02:40] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ruthgavi was modified, changed by Ruthgavi link https://wikitech.wikimedia.org/w/index.php?diff=169739 edit summary: 
[13:05:08] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ruthgavi was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=169740 edit summary: As far as I can see, you have access to tool labs.
[13:12:17] <Ruthgavi>	 Hi
[13:12:28] <valhallasw`cloud>	 Ruthgavi: hey.
[13:12:40] <Ruthgavi>	 I am having a problem connecting to tools-login.wmflabs.org
[13:13:07] <Ruthgavi>	 I had access to it in 2014 from another computer
[13:13:38] <Ruthgavi>	 now I have the error: Permission denied (publickey,hostbased)
[13:13:58] <valhallasw`cloud>	 Ruthgavi: Jul  9 13:10:08 tools-bastion-01 sshd[27133]: input_userauth_request: invalid user Ruthgavi [preauth]
[13:14:11] <valhallasw`cloud>	 your shell username is ruthgavi (without capital R)
[13:14:26] <Ruthgavi>	 oh
[13:14:28] <Ruthgavi>	 ok
[13:14:32] <Ruthgavi>	 let me try again
[13:14:55] <Ruthgavi>	 YOu are right
[13:14:58] <Ruthgavi>	 thanks a lot! 
[13:15:02] <Ruthgavi>	 :)
[13:15:14] <Ruthgavi>	 Have a good day
[13:15:18] <Ruthgavi>	 bye
[13:15:35] <valhallasw`cloud>	 you're welcome!
[13:35:09] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] Passwords/keys for Nodepool (031 comment) [labs/private] - 10https://gerrit.wikimedia.org/r/223536 (https://phabricator.wikimedia.org/T89143) (owner: 10Hashar)
[13:35:37] <grrrit-wm>	 (03PS3) 10Hashar: Passwords/keys for Nodepool [labs/private] - 10https://gerrit.wikimedia.org/r/223536 (https://phabricator.wikimedia.org/T89143) 
[13:37:08] <grrrit-wm>	 (03PS4) 10Hashar: Passwords/keys for Nodepool [labs/private] - 10https://gerrit.wikimedia.org/r/223536 (https://phabricator.wikimedia.org/T89143) 
[13:37:24] <grrrit-wm>	 (03CR) 10Hashar: "Added a dummy modules/nodepool/files/dib_jenkins_id_rsa" [labs/private] - 10https://gerrit.wikimedia.org/r/223536 (https://phabricator.wikimedia.org/T89143) (owner: 10Hashar)
[13:43:38] <wikibugs>	 6Labs, 10Tool-Labs, 7Database, 3Labs-Q4-Sprint-1, and 4 others: Make sure tools-db is replicated somewhere - https://phabricator.wikimedia.org/T88718#1441209 (10jcrespo) @yuvipanda @Coren The estimated downtime needed is 10 minutes, with some minor degradations by extra use of disk and network before and a...
[13:48:54] <wikibugs>	 6Labs, 3Labs-Sprint-105: Do a manual backup of labstore1002 - https://phabricator.wikimedia.org/T104882#1441221 (10Andrew) https://etherpad.wikimedia.org/p/lvm-labstore-backups
[13:50:48] <wikibugs>	 6Labs, 10Tool-Labs, 7Database, 3Labs-Q4-Sprint-1, and 4 others: Make sure tools-db is replicated somewhere - https://phabricator.wikimedia.org/T88718#1441234 (10yuvipanda) Ideally this should happen post Wikimania, so after 20th July...  Or it can happen on Monday if @Coren is ok with doing that? A bit of...
[13:58:52] <wikibugs>	 6Labs, 10Tool-Labs, 7Database, 3Labs-Q4-Sprint-1, and 4 others: Make sure tools-db is replicated somewhere - https://phabricator.wikimedia.org/T88718#1441256 (10jcrespo) My suggestion: let's wait after Wikimania, and give a lot of advance notice (but you, Coren and Yuvi are the bosses here).  If you feel o...
[14:01:46] <wikibugs>	 6Labs, 10Tool-Labs, 7Database, 3Labs-Q4-Sprint-1, and 4 others: Make sure tools-db is replicated somewhere - https://phabricator.wikimedia.org/T88718#1441264 (10coren) @yuvipanda: Yeah, Monday is not enough notice, even if the outage is expected to be brief.  Let's schedule that for after our return from W...
[14:07:28] <wikibugs>	 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103, 3Labs-Sprint-104, 3Labs-Sprint-105: Labs: increase size of the volume for the maps project and restore - https://phabricator.wikimedia.org/T103358#1441272 (10coren) This is nearing completion, with only some parts of `wma/` left to be restored....
[14:09:56] <Coren>	 YuviPanda: Are the 20150703 complete?  Your last update on the ticket says ongoing but I'm not seeing a running process?
[14:10:09] <Coren>	 20150703 *backups
[14:10:16] <YuviPanda>	 Coren: whoops, yes, they completed a couple of days ago...
[14:10:24] <YuviPanda>	 monday, actually
[14:10:32] <YuviPanda>	 I had told mar.k but forgot to put it on the ticket.
[14:10:40] <Coren>	 kk; ima blow up the snapshots now so I can start testing the automated version.
[14:11:07] <Coren>	 Just wanted to ask you first before I start throwing lvremoves around.  :-)
[14:11:27] <YuviPanda>	 Coren: ok! Can you do the automated ones in python3 as well?
[14:11:37] <YuviPanda>	 instead of python2?
[14:11:42] <YuviPanda>	 there shouldn't be much of a difference
[14:11:48] <YuviPanda>	 I can always port it forward as well if you'dl ike
[14:12:26] <YuviPanda>	 Coren: also, do you think you can babysit a manual run of replica-addusers.pl? 
[14:12:30] <YuviPanda>	 it's been broken for a few weeks now...
[14:12:41] <Coren>	 YuviPanda: I've no objection, but remember I'm a python newb.  This is like my fifth bit of code in python at all so I don't even know if I've got python2 vs python3 things in it.  :-)
[14:12:52] <YuviPanda>	 Coren: :) 'tis ok. is there already a patch up?
[14:15:14] <Coren>	 YuviPanda: The original code was abandonned a while back as it presumed thin volumes and kept a whole set of volumes, I've got some ugly poc I want to try first then I'll push to gerrit if it works at all.  The code is based on is the one in https://gerrit.wikimedia.org/r/#/c/199267/ but there is only one program now.  I expect I'll have something up for review in about 2h
[14:15:32] <YuviPanda>	 alright, cool!
[14:16:21] <Coren>	 Still learning python though, do you have a primer on the biggest differences between 2 and 3 I should be aware of?
[14:16:49] <YuviPanda>	 Coren: I think for scripty code like this there isn't really much of a difference...
[14:16:58] <YuviPanda>	 print is a function and not a statement might be the biggest one :)
[14:23:49] <Coren>	 I'm pretty sure I've never relied on print having a return value or not.  :-)
[14:24:51] <Coren>	 Also, I don't mind babysitting a run of replica-addusers, but I would have expected we didn't want to add more old-style grants right now?
[14:25:07] <andrewbogott>	 Coren, YuviPanda|lunch, does this mean I shouldn’t do the by-hand backup that I was about to start?
[14:25:46] <Coren>	 andrewbogott: It might be better not to, if only because if everything works right a side effect of my final test will be a backup.  :-)
[14:25:56] <andrewbogott>	 ok!
[14:26:03] <Coren>	 andrewbogott: From a wasted effort pov
[14:26:27] <Coren>	 My first couple tests only does very limited selection of copies though, so it will delay the full backup for a bit.
[14:26:40] <Coren>	 If a couple extra hours isn't an issue, then don't bother.
[14:28:27] <Coren>	 BTW, ma.rk mentionned a cheat cheat he's given you you are working from, but I didn't ask him where it was? 
[14:28:47] <andrewbogott>	 https://etherpad.wikimedia.org/p/lvm-labstore-backups
[14:30:00] <wikibugs>	 6Labs, 5Patch-For-Review, 7Tracking: Make OpenStack Horizon useful for production labs - https://phabricator.wikimedia.org/T87279#1441363 (10Andrew)
[14:30:02] <wikibugs>	 6Labs, 3Labs-Sprint-103, 3Labs-Sprint-104, 3Labs-Sprint-105: Limit available images on horizon - https://phabricator.wikimedia.org/T91782#1441361 (10Andrew) 5Open>3stalled Getting this into the upstream properly requires a lot of scary javascript -- to write that I'll need a better test platform than I...
[14:30:29] <Coren>	 andrewbogott: ty
[14:31:25] <YuviPanda|lunch>	 Coren: nope! https://gerrit.wikimedia.org/r/#/c/223781/
[14:32:16] <Coren>	 YuviPanda|lunch: Oh, allright - though that means new users won't be able to create new dbs yet.  Probably not an issue.
[14:32:23] <YuviPanda|lunch>	 yup
[14:33:59] <Coren>	 e43dad8dd2ef2b7a8506d017b2f317c1a5078b82 doesn't look like a legit user to me.
[14:36:09] <jynus>	 oh, I have a test user over there called u10000000000000000000 or something, hope you don't mind
[14:37:01] <Coren>	 jynus: Nope.  Just noticed a funky "real" one, not your tests.  :_)
[14:40:52] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1441395 (10scfc) I don't mind if ops wouldn't have investigated the issue.  But doing so, claiming...
[14:48:53] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1441425 (10coren) >>! In T92351#1441395, @scfc wrote: > But doing so, claiming that `dnsmasq` is at...
[14:48:58] <wikibugs>	 6Labs: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1441426 (10Andrew) This is the quick revert we did when ripping out nutcracker broke things:  https://gerrit.wikimedia.org/r/#/c/158238/1/wmf-config/CommonSettings.php
[14:56:22] <Hazard-SJ>	 Hi, do I have to manually install SSL support in my virtualenv?
[14:57:10] <valhallasw`cloud>	 no....?
[14:57:35] <Hazard-SJ>	 class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):
[14:57:37] <Hazard-SJ>	 AttributeError: 'module' object has no attribute 'HTTPSConnection'
[14:58:41] <valhallasw`cloud>	 that doesn't actually tell me much more :-p
[14:58:46] <valhallasw`cloud>	 what's the entire stacktrace?
[15:00:16] <Hazard-SJ>	 http://pastebin.com/zzNqQ70d
[15:00:28] <Hazard-SJ>	 And that's only when I submit it as a job
[15:00:48] <Hazard-SJ>	 If I run it directly in tools-login, it works fine
[15:01:48] <valhallasw`cloud>	 ah. You're probably not using the python in your venv, but rather the system-wide python
[15:01:56] <valhallasw`cloud>	 which is 2.6 on precise hosts and 2.7 on trusty hosts
[15:02:19] <valhallasw`cloud>	 but given that your venv is 2.7, you'll need to schedule on trusty in any case
[15:02:31] <wikibugs>	 6Labs: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1441468 (10Andrew) Simply removing mc.php results in  [Thu Jul 09 15:01:49.276847 2015] [:error] [pid 1496] [client 2620:0:861:119:d6ae:52ff:fe76:4515:38897] PHP Notice:  Undefined variable: endpointURL in /srv/mediawiki/php-1.26w...
[15:02:52] <valhallasw`cloud>	 jsub --release=trusty /data/project/hazard-bot/pywikibot/bin/python <script to python script>
[15:02:54] <valhallasw`cloud>	 should work
[15:03:35] <Hazard-SJ>	 Okay, I'll try that
[15:07:20] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 10Labs-Infrastructure: Reenable resolv.conf ndots:2 option on CI instances - https://phabricator.wikimedia.org/T105297#1441491 (10Dzahn) I have abandoned https://gerrit.wikimedia.org/r/#/c/196731/ because Coren said "This was the wrong workaround for a problem...
[15:09:04] <Hazard-SJ>	 valhallasw`cloud: This --release=trusty thing is preventing the job from going through, it's showing usage now
[15:10:38] <valhallasw`cloud>	 Hazard-SJ: sorry, -l release=trusty
[15:11:55] <grrrit-wm>	 (03PS1) 10Sitic: Show a edit diff when the user requests it [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/223798 (https://phabricator.wikimedia.org/T103679) 
[15:14:17] <Hazard-SJ>	 Seems to work, thanks, I'll test that out with my other tools and adjust them if necessary
[15:14:59] <grrrit-wm>	 (03CR) 10Sitic: [C: 032 V: 032] Show a edit diff when the user requests it [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/223798 (https://phabricator.wikimedia.org/T103679) (owner: 10Sitic)
[15:19:00] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6Security, 7Security-Other: Huge rash of bot accounts on wikitech - https://phabricator.wikimedia.org/T105350#1441534 (10coren) 3NEW
[15:29:59] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6Security, 10wikitech.wikimedia.org, 7Security-Other: Huge rash of bot accounts on wikitech - https://phabricator.wikimedia.org/T105350#1441577 (10Krenair) Didn't we just turn on a bunch of anti-abuse measures?
[15:32:07] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1441594 (10Krenair)
[15:49:17] <Krenair>	 Coren, YuviPanda|lunch: you guys want those osm reverts deployed now?
[15:49:30] <Krenair>	 YuviPanda, ^
[15:49:30] <YuviPanda>	 Krenair: no, I don't think they should be deployed
[15:49:31] <YuviPanda>	 wait
[15:49:39] <Coren>	 Krenair: Can wait until swat
[15:49:44] <Krenair>	 We're in swat now
[15:49:46] <Krenair>	 That's why I'm asking
[15:50:03] <YuviPanda>	 Coren: I don't think it shoudl be deployed at all?
[15:52:13] <Huji>	 Hi everyone. What is the best way to set up a MW wiki on Labs?
[15:52:27] <Huji>	 I need it for testing purposes only (traffic will be from me and a few other technical users only)
[15:53:23] <Nikerabbit>	 I heard there is some way to just use the vagrant there
[15:53:26] <andrewbogott>	 Huji: typically people use labs-vagrant these days.
[15:53:33] <andrewbogott>	 But of course you need a project and an instance and such.
[15:53:45] <Huji>	 So I have a project I believe
[15:54:01] <Huji>	 I am very poorly educated with Labs lingo
[15:54:30] <Huji>	 andrewbogott: but I think my project is also call 'Huji'
[15:54:52] <Krenair>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Huji - nope..
[15:54:57] <andrewbogott>	 Huji: I don’t see evidence that such a project exists
[15:55:16] <Huji>	 Oh well, I at least have an account and can run bot sciprts and such
[15:55:36] <Huji>	 and I remember a long time ago I requested a project to be created; may be it was removed due to inactivity? Do you guys do that?
[15:56:13] <andrewbogott>	 Possibly, if it was very very old.
[15:56:34] <Huji>	 not that old, like months
[15:56:34] <Huji>	 I think
[15:56:47] <Huji>	 andrewbogott: at any rate; let me tell you the intention:
[15:57:00] <Huji>	 I want to set up a MW wiki and install Twinkle on it, and then translate Twinkle there
[15:57:15] <Huji>	 and test it and such, before rolling it out on a Wikimedia Wiki
[15:57:25] <Huji>	 would I need a separate project for that?
[15:57:39] <Huji>	 or can I just install MW in my environment as is?
[15:57:40] <andrewbogott>	 Huji: is there a reason to do that on labs rather than locally using vagrant?
[15:57:55] <Huji>	 I can do locally, if that is the best idae
[15:58:09] <Huji>	 so long as I can access it from the web and have a few other people help me with translation and testing
[15:58:23] <andrewbogott>	 oh, that’s not what I meant by ‘locally’ :)
[15:58:39] <andrewbogott>	 If you need other people to access it, that’s a good reason not to install the wiki just on your laptop
[15:59:11] <Huji>	 andrewbogott: yeah, locally on my laptop is not an option!
[15:59:13] <andrewbogott>	 Huji: step one is to either locate the project you already have (?) or request a new one.  https://phabricator.wikimedia.org/T76375
[15:59:18] <grrrit-wm>	 (03PS1) 10Sitic: Cleanup css files [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/223811 
[15:59:28] <andrewbogott>	 by adding a subtask to that phab case
[15:59:31] <grrrit-wm>	 (03CR) 10Sitic: [C: 032 V: 032] Cleanup css files [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/223811 (owner: 10Sitic)
[16:02:11] <Huji>	 andrewbogott: on it
[16:02:31] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/WMDE-leszek was created, changed by WMDE-leszek link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/WMDE-leszek edit summary: Created page with "{{Tools Access Request |Justification=I will be contributing to tools developed at Wikimedia Deutschland e.V. I am working at Wikimedia Deutschland since June 2015 |Completed=..."
[16:04:42] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441673 (10Huji) 3NEW
[16:05:13] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441683 (10Huji)
[16:05:52] <Huji>	 andrewbogott: request has been made
[16:05:58] <Huji>	 I assume next steps are ... https://wikitech.wikimedia.org/wiki/Help:Labs-vagrant
[16:06:13] <Huji>	 is that correct?
[16:06:49] <andrewbogott>	 Huji: approximately.  You’ll also need to create an instance in the project.
[16:07:20] <Huji>	 Right. I think that page also says that.
[16:07:40] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441698 (10Andrew) I believe that projects need to start with a lower-case letter, and cannot have spaces.  faWP ok?
[16:08:51] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441704 (10yuvipanda) Why can this not run on tool labs?
[16:09:07] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441705 (10Huji) Andrew faWP is good, even fa.wp or fa-wp works. Please use the same naming scheme that most other projects use (I think it is hyphenated?)
[16:09:55] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441713 (10Andrew) Yuvi -- I believe that Huji wants to set up a private wiki as a translation hub.
[16:10:49] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441714 (10Huji) >>! In T105356#1441704, @yuvipanda wrote: > Why can this not run on tool labs?  Well part of our work can. As I said we do a lot of bots, and we are using tool labs for that. However, the id...
[16:11:05] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441715 (10Huji) >>! In T105356#1441713, @Andrew wrote: > Yuvi -- I believe that Huji wants to set up a private wiki as a translation hub.  That too!
[16:12:47] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441724 (10yuvipanda) Having your own project would imply that you're going to have to do a fair bit of sysadminy things yourself. If you're ok with that, then I guess I'm ok having this project created :)
[16:14:55] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1441736 (10Huji) Yes that's fine.
[16:39:26] <physikerwelt>	 hi I updated labs-vagrant to the most recent mw version and get a dabase error now. However I can run mysql from the console and don't see any issues http://drmf-beta.wmflabs.org/
[16:39:33] <physikerwelt>	 Any ideas?
[16:40:34] <bd808>	 physikerwelt: hmmm... this looks like a config problem that I thought we fixed quite a while ago for mediawiki-vagrant
[16:40:42] * bd808 looks for patch
[16:41:19] <physikerwelt>	 I ran labs-vagrant git-update should that not update vagrant as well?
[16:41:29] <bd808>	 ah. no it does not
[16:41:38] <physikerwelt>	 ok sorry
[16:41:40] <bd808>	 try a git pull in /vagrant
[16:42:26] <bd808>	 physikerwelt: I think this is the config change you are missing -- https://gerrit.wikimedia.org/r/#/c/216120/
[16:42:41] <physikerwelt>	 bd808: thank you it's already solved
[16:42:56] <marxarelli>	 bd808: maybe we should pair on creating a deb for vagrant 1.7 at the hackathon
[16:43:02] <physikerwelt>	 I was not aware that git-update did not updata vagrant itself
[16:43:20] <marxarelli>	 to pave the way for mw-vagrant lxc glory
[16:44:12] <bd808>	 marxarelli: I think we really just need https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.2_x86_64.deb imported into our apt repo
[16:44:47] <bd808>	 we get elasticsearch and logstash debs that way (downloading the deb from upstream)
[16:45:06] <marxarelli>	 backporting the package from sid shouldn't be that hard, though
[16:45:22] <bd808>	 true
[16:45:41] <bd808>	 my deb fu is a little rusty but I'd be willing to pitch in
[16:46:24] <YuviPanda>	 you should file a bug if there isn't one already
[16:46:35] <bd808>	 there's been one filred for ages
[16:46:39] <marxarelli>	 YuviPanda: there is one
[16:46:56] <marxarelli>	 i think the cookie got licked but has yet to be eaten
[16:46:57] <bd808>	 https://phabricator.wikimedia.org/T93153
[17:03:11] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442047 (10yuvipanda) Should allow people to:  # ssh on (we still support agent forwarding, unfortunately) # ssh on via proxycommand
[17:11:15] <wikibugs>	 6Labs, 10Labs-Infrastructure, 6Security, 10wikitech.wikimedia.org, 7Security-Other: Huge rash of bot accounts on wikitech - https://phabricator.wikimedia.org/T105350#1442076 (10yuvipanda) We aren't granting shell automatically - see T105353.
[17:23:45] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442155 (10scfc) We could also utilize `/etc/cron.allow`/`/etc/cron.deny`.  However, I'm not sure if they affect Puppet, i. e. if an empty `/etc/cron.allow` prevents Puppet from setting up...
[17:25:12] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442163 (10yuvipanda) Yeah but I was hoping to be purely whitelist based, similar to old sillyshell - a small python script that laughs at you if you try to do anything else outside of ssh...
[17:30:33] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442172 (10scfc) A custom, restricted shell would certainly be preferable as it prevents (most) other abuses as well.
[17:40:28] <Steinsplitter>	 hi
[17:40:29] <Steinsplitter>	 i get   _mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query')
[17:40:38] <Steinsplitter>	 when executing verry large querys
[17:40:59] <Steinsplitter>	 mariadb is closing connections without a reason.
[17:43:06] * Steinsplitter pokes Coren
[17:43:42] <Coren>	 Steinsplitter: It's not without reason, certainly.  You're probably hitting against a time or memory limit.
[17:43:52] <Coren>	 They are very liberal, but they are there.  What is the query?
[17:44:51] <Steinsplitter>	 Coren: http://pastebin.com/GA4Au4E2   it has woked vor a verry long time without problems
[17:45:03] <Steinsplitter>	 maybe there has been added some sort of limit in the last days?
[17:46:05] <Coren>	 jynus: ^^ anything recent that might be the cause?
[17:47:01] <jynus>	 ipblocks has been changed upstream
[17:47:07] <jynus>	 on production
[17:49:12] <Steinsplitter>	 jyuns: i checked it, all table are existing and the syntacks is okay. for some reason the db is ding after 30 minutes
[17:51:04] <jynus>	 I can see your query running, I have not implemented any hard limit
[17:53:24] <jynus>	 I will investigate, 1002 saw corruption in the past, could be related
[17:54:07] <Steinsplitter>	 ok, thx :)
[18:52:53] <hashar>	 andrewbogott: hey :)  I managed to get the Jessie image to boot!
[18:53:03] <andrewbogott>	 great!
[18:53:04] <wikibugs>	 6Labs: Delete SuchABot Account - https://phabricator.wikimedia.org/T105384#1442511 (10yuvipanda) 3NEW
[18:53:15] <hashar>	 was missing ifconfig and a dhpclient, that caused some hidden exception/error in cloudinit
[18:53:18] <hashar>	 took me a while 
[18:53:19] <andrewbogott>	 So it turned out to be a change in the default behavior of the image builder?
[18:53:31] <hashar>	 I might never tried building a Jessie image
[18:53:37] <hashar>	 the previous attempts were with Ubuntu Trusty
[18:53:40] <andrewbogott>	 ah, makes sense :)
[18:53:49] <hashar>	 and the disk image element to build an ubuntu machine does include a bunch of packages
[18:53:55] <hashar>	 I guess upstream (openstack) barely uses debian
[18:54:05] <hashar>	 I have sent patches upstream
[18:55:06] <andrewbogott>	 yeah, it’s confusing.  The Designate people mostly use debian but I’m not clear that the other components are well-supported there.
[18:55:12] <andrewbogott>	 I will have to learn all this sometime soon :(
[18:55:24] <hashar>	 I can probably share my experience :}
[18:55:50] <hashar>	 the openstack deb packages are maintained by Thomas "zigo"
[18:55:56] <hashar>	 he might be able to help if needed
[18:56:08] <hashar>	 I think he recently backported / updated openstack in Jessie
[18:56:24] <wikibugs>	 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1442531 (10Andrew) 5Open>3Resolved a:3Andrew Project fa-wp created.  Please open a cleanup bug if/when you are finished with the project so we can reclaim resources.
[18:56:25] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1442534 (10Andrew)
[18:56:26] <hashar>	 and overall he is very prompt to update .deb packages when a bug is filled
[18:56:46] <andrewbogott>	 Ah, yeah, he’s the one I was thinking of who built the debian designate stuff.
[18:57:18] <hashar>	 he would certainly be happy to hear some feedback
[19:01:03] <hashar>	 andrewbogott: also I replied with the private pass/api keys/ ssh key etc for nodepool. Hopefully it is clear enough
[19:01:18] <andrewbogott>	 hashar: yep, I saw, I will try to catch up with that shortly.
[19:01:28] <hashar>	 it is not a blocker anyway
[19:01:42] <hashar>	 I made a point to port whatever I did on the machine so the puppet patch should be all fine
[19:09:16] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442552 (10Andrew) Are there no pre-existing minimal shells that we can adopt rather than writing our own?
[19:14:58] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442566 (10yuvipanda) There's `rbash` but that's not restricted enough...
[19:15:24] <grrrit-wm>	 (03PS5) 10Hashar: Passwords/keys for Nodepool [labs/private] - 10https://gerrit.wikimedia.org/r/223536 (https://phabricator.wikimedia.org/T89143) 
[19:17:48] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032 V: 032] Passwords/keys for Nodepool [labs/private] - 10https://gerrit.wikimedia.org/r/223536 (https://phabricator.wikimedia.org/T89143) (owner: 10Hashar)
[19:18:29] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442587 (10valhallasw) I think the typical method is to drop people in a chroot jail:  http://www.techrepublic.com/blog/linux-and-open-source/chroot-users-with-openssh-an-easier-way-to-con...
[19:27:47] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442608 (10hashar) What I have set up once upon a time, is the ssh connection to the bastion landed in a chroot which had the very minimal commands available. Namely at the time Kerberos t...
[19:30:15] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442609 (10yuvipanda) I think a chroot is more than what we need, since we don't want to restrict users to just their homedirs - we want to restrict them only by the commands they can run....
[19:33:20] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442611 (10Andrew) Something like this is an option:  http://cybermashup.com/2013/05/14/restrict-ssh-logins-to-a-single-command/  Would it also be possible to just configure systems so tha...
[19:59:08] <wikibugs>	 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103, 3Labs-Sprint-104, 3Labs-Sprint-105: Labs: increase size of the volume for the maps project and restore - https://phabricator.wikimedia.org/T103358#1442725 (10coren) The maps project's NFS have been restored, but it is almost certain that a remou...
[20:28:30] <wikibugs>	 6Labs, 6Discovery, 10Maps: WikiMiniAtlas (wma.wmflabs.org) is still down - https://phabricator.wikimedia.org/T104417#1442854 (10coren)
[20:28:32] <wikibugs>	 6Labs, 6Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1442855 (10coren)
[20:28:32] <wikibugs>	 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-103, 3Labs-Sprint-104, 3Labs-Sprint-105: Labs: increase size of the volume for the maps project and restore - https://phabricator.wikimedia.org/T103358#1442852 (10coren) 5stalled>3Resolved The instances of the maps project have been rebooted, and...
[20:28:44] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1442860 (10Ankita-ks)
[20:29:08] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#798989 (10Ankita-ks)
[20:30:10] <Coren>	 andrewbogott_afk: I got the new process running as a test over the tools filesystem now.  I'll be back later tonight to check up on it and start the other two filesystems if all goes well.
[20:30:35] <Coren>	 YuviPanda (also) ^^
[20:30:50] <Coren>	 In the meantime, I have to go for groceries.
[20:34:04] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442891 (10valhallasw) sillyshell is problematic for Windows users in the same way ProxyCommand is (although it's slightly less problematic). This would require windows users to find the r...
[20:38:53] <wikibugs>	 6Labs, 5Patch-For-Review: Set a custom shell for people on the bastion project - https://phabricator.wikimedia.org/T102395#1442909 (10yuvipanda) Oh yeah - not silly shell itself but something similar. I'll start experimenting with a different project and see how it goes :) I do want to make sure agent forwardi...
[20:46:38] <valhallasw`cloud>	 YuviPanda: please test with putty specifically :-p
[20:47:02] <valhallasw`cloud>	 it's not so much about agent forwarding, but rather about 'help how do I use this option if I don't have .ssh/config or a command line'
[21:15:32] <YuviPanda>	 valhallasw`cloud: yeah agreed.
[21:15:48] <YuviPanda>	 I don't think a replacement that users notice is viable 
[21:15:59] <YuviPanda>	 Should not require any changes in access method I guess 
[21:19:44] <valhallasw`cloud>	 YuviPanda: maybe we should provide a preconfigured putty somehow...
[21:21:00] <YuviPanda>	 valhallasw`cloud: oooh maybe 
[21:21:07] <YuviPanda>	 A download this and it will do the right thing 
[21:21:55] <valhallasw`cloud>	 *nod* maybe we can even do proxy command then
[21:22:23] <valhallasw`cloud>	 Hm. mosh will need a chroot
[21:22:38] <YuviPanda>	 We can whitelist the stuff mosh does 
[21:22:46] <YuviPanda>	 Actually maybe we should start with a logger shell
[21:22:55] <YuviPanda>	 Logs commands and passes through to bash 
[21:23:06] <YuviPanda>	 And use it to test and figure out different methods of access 
[21:23:30] <valhallasw`cloud>	 Mm. Let me sleep on that
[21:24:42] <YuviPanda>	 OK :)
[21:35:52] <wikibugs>	 6Labs: can't import _SSL in /usr/lib/python2.7/ssl.py - https://phabricator.wikimedia.org/T105403#1443162 (10Capt_Swing) 3NEW
[21:36:40] <fhocutt>	 ^ YuviPanda, valhallasw`cloud 
[21:37:31] <fhocutt>	 any thoughts? This is really weird.
[21:43:50] <YuviPanda>	 fhocutt: try -l release=trusty with jsub
[21:44:30] <fhocutt>	 so jsub -N <name> -l release=trusty <command>?
[21:44:39] <YuviPanda>	 Yes
[21:45:25] <fhocutt>	 trying it, ty
[21:45:49] <wikibugs>	 6Labs: can't import _SSL in /usr/lib/python2.7/ssl.py - https://phabricator.wikimedia.org/T105403#1443175 (10Fhocutt) p:5Triage>3High
[22:05:24] <wikibugs>	 6Labs: can't import _SSL in /usr/lib/python2.7/ssl.py - https://phabricator.wikimedia.org/T105403#1443230 (10Capt_Swing) Fixed by @yuvipanda and @fhocutt. Solution below.   ```  jsub -N <name> -l release=trusty <command> ```
[22:05:36] <wikibugs>	 6Labs: can't import _SSL in /usr/lib/python2.7/ssl.py - https://phabricator.wikimedia.org/T105403#1443232 (10Capt_Swing) 5Open>3Resolved
[22:10:46] <fhocutt>	 YuviPanda, that did it
[22:10:48] <fhocutt>	 thanks very much
[22:15:50] <fhocutt>	 YuviPanda, actually, could you tell me why that worked?
[22:19:02] <YuviPanda>	 fhocutt: oh yes
[22:19:19] <YuviPanda>	 fhocutt: by default jsub runs jobs in ubuntu precise, even though tools-login is ubuntu trusty
[22:19:29] <YuviPanda>	 this we have to change but do not know how to in a backwards compatible way...
[22:19:39] <fhocutt>	 .......
[22:19:41] <fhocutt>	 ok
[22:19:45] <fhocutt>	 computers!
[22:20:05] <YuviPanda>	 yes unfortunately
[22:20:18] <YuviPanda>	 so yes, jsub -l release=trusty until then :)
[22:20:25] * YuviPanda waves off now, have fun
[22:20:30] <fhocutt>	 do you have any idea how it is that this cron thing worked earlier this year and not now?
[22:20:42] <fhocutt>	 or I will ask you later :)