[00:23:38] <wikibugs>	 6Labs, 10Tool-Labs, 10Diffusion, 10Internet-Archive: Copy contents of https://svn.toolserver.org/ to Wikimedia Diffusion - https://phabricator.wikimedia.org/T60801#1954260 (10mmodell) looks like that's going to work. Currently importing the dumps into a new empty repo.
[00:59:34] <darkblue_b>	 whoa hey rise-up email - okay
[00:59:58] <YuviPanda>	 ?
[01:01:45] <darkblue_b>	 hi YuviPanda - g'day to you
[01:01:50] <darkblue_b>	 Brian from OSGeo here
[01:02:26] <darkblue_b>	 nothing major - I have lived in the Bay Area a long time.. met a wide variety of folks along the way
[01:02:39] <darkblue_b>	 couldnt help but notice
[01:03:05] <darkblue_b>	 however one variety of folks I did not meet was -labs people at WMF HQ.. oh well 
[01:03:27] <darkblue_b>	 no secret - you are miles ahead of where we are at OSGeo.. no worries .. maybe the 20th 
[01:15:58] <Dispenser>	 Despite my computer bluescreening/power failure/network outage/etc people keep telling me I have better uptime than Labs
[01:15:59] <Dispenser>	 This is absurd, because its connected (SSH Tunnel) to the Labs DBs.  Fix webservice.
[01:19:37] <YuviPanda>	 hey Dispenser
[01:19:53] <YuviPanda>	 actually reporting what problem you are encoutering would be actually helpful
[01:20:56] <Dispenser>	 Some PHP script that I'm not a maintainer of (but links to my tools) keeps dying.  But the quote from somebody unrelated me.
[01:21:11] <YuviPanda>	 'some PHP script' does not help narrow anything down
[01:21:25] <Dispenser>	 https://tools.wmflabs.org/dplbot/dab_fix_list.php
[01:21:54] <Dispenser>	 I've asked to become a maintainer (so I hit the reboot button)
[01:21:59] <YuviPanda>	 the webservice for that seems to be up?
[01:22:27] <YuviPanda>	 we rebooted all our machines earlier today to deal with a kernel security vulnerability
[01:22:44] <Dispenser>	 s/reboot/restart/
[01:22:47] <YuviPanda>	 so there were probably not working around that time
[01:23:03] <YuviPanda>	 this was announced yesterday and today on the labs-announce list
[01:23:24] <YuviPanda>	 if you aren't subscribed, please do subscribe.
[01:25:06] <Dispenser>	 webservice seems to crash often for this tool.  Then the watchdog script brings it back <1 hour later
[01:26:31] <YuviPanda>	 indeed, it's been restarted 23759 times
[01:26:33] <YuviPanda>	 since the beginning
[01:26:47] <YuviPanda>	 if it's hitting the memory limit and crashing
[01:26:50] <YuviPanda>	 there's very little we can do
[01:27:08] <YuviPanda>	 it should restart in about 20s though
[01:27:09] <YuviPanda>	 not <1h
[01:27:16] <YuviPanda>	 so if you can point to a specific time period
[01:27:18] <YuviPanda>	 I can investigate
[01:30:04] <Dispenser>	 YuviPanda: Go knock yourself out.  https://en.wikipedia.org/wiki/User_talk:Dispenser/Dablinks#Dablinks_not_working
[01:30:20] <YuviPanda>	 I think if you don't change your tone Dispenser
[01:30:24] <YuviPanda>	 I'll have to put you on ignore
[01:30:36] <Reedy>	 YuviPanda: That's not an insult
[01:30:39] <YuviPanda>	 that time period is during our outage
[01:30:56] * YuviPanda verifies that
[01:31:02] <Reedy>	 "Have at it"
[01:31:04] <Reedy>	 "glhf"
[01:33:14] <YuviPanda>	 yeah
[01:33:17] <YuviPanda>	 definitely during the planned outage
[01:33:20] <YuviPanda>	 so
[01:33:22] <YuviPanda>	 *shrug*
[04:11:01] <wikibugs>	 6Labs, 7Tracking: Create a Labs project for video2commons - https://phabricator.wikimedia.org/T124387#1954583 (10zhuyifei1999) 3NEW
[04:13:10] <wikibugs>	 6Labs, 7Tracking: Create a Labs project for video2commons - https://phabricator.wikimedia.org/T124387#1954592 (10zhuyifei1999) If possible, having 1 instance of m1.gigantic would be awesome for the transcoding part.
[08:45:14] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#1954871 (10Matanya)
[08:45:16] <wikibugs>	 6Labs, 7Tracking: Create a Labs project for video2commons - https://phabricator.wikimedia.org/T124387#1954868 (10Matanya) 5Open>3Resolved a:3Matanya Done on encoding01, no need for a new project. Thanks for your work!
[12:16:22] <marmick>	 hi YuviPanda! I just checked a process I was running in Labs and it stopped yesterday at 18.07. I read the channel topical and I understood it was due to the reboots. The process takes about 10 days. Should I start it again?
[12:45:40] <Reedy>	 marmick: Yeah, you should be fine to do it
[12:45:51] <Reedy>	 The reboots were necessary due to kernel bugs that needed addressing
[13:29:52] <marmick>	 Reedy: yes, i understand. but it already happened the January 3rd or so... so I would not like to happen again. my script it takes 10 days
[13:30:08] <Reedy>	 I guess the answer is tough
[13:30:17] <marmick>	 but at this pace...it might take 5 weeks or more :(
[13:30:21] <Reedy>	 Guaranteeing uptime is almost impossible
[13:30:23] <marmick>	 unpredictable
[14:20:21] <grrrit-wm>	 (03CR) 10Luke081515: [C: 031] Add PLURAL support to LOGEVENT_RENAMEUSER message [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265647 (https://phabricator.wikimedia.org/T114876) (owner: 10MtDu)
[14:21:33] <grrrit-wm>	 (03CR) 10MGChecker: [C: 031] Add PLURAL support to LOGEVENT_RENAMEUSER message [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265647 (https://phabricator.wikimedia.org/T114876) (owner: 10MtDu)
[14:46:42] <grrrit-wm>	 (03CR) 10MtDu: "Thanks for the reviews! Does any one have +2 power? Seems like this is approved." [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265647 (https://phabricator.wikimedia.org/T114876) (owner: 10MtDu)
[15:17:07] <grrrit-wm>	 (03PS1) 10Hashar: Introduce tox as an entry point [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265735 
[15:18:50] <tonythomas>	 Can somebody take a quick look into http://newsletter-test.wmflabs.org/ ?
[15:19:43] <tonythomas>	 its showing a 502 gateway error, and I get the error on running vagrnat up with vagrant user in the labs instance : 
[15:19:44] <tonythomas>	 "Message: RuntimeError: The mediawiki-vagrant plugin hasn't been installed yet. Please run `setup.sh`."
[15:20:16] <tonythomas>	 and ./setup.sh shows https://dpaste.de/w8H9/raw
[15:21:32] <grrrit-wm>	 (03CR) 10Hashar: "check experimental" [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265735 (owner: 10Hashar)
[15:29:02] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Create a larger newsletter-test instance in labs - https://phabricator.wikimedia.org/T120516#1955520 (1001tonythomas) Just now got this new error:   ``` mwvagrant@newsletter-test:/srv/mediawiki-vagrant$ source /etc/profile.d/alias-vagrant.sh mwvagrant@newsletter-test:/...
[15:41:56] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Create a larger newsletter-test instance in labs - https://phabricator.wikimedia.org/T120516#1955552 (10valhallasw) Presumably that was fixed in  https://github.com/wikimedia/mediawiki-vagrant/commit/ea2ffc3034caa2526d2601cd96fe1fc659907271  so updating to the latest m...
[15:42:29] <wikibugs>	 10Tool-Labs-tools-Global-user-contributions, 7Easy, 7JavaScript: GUC counter doesn't increment - https://phabricator.wikimedia.org/T123879#1955553 (10Danny_B) p:5Triage>3Low
[15:54:09] <bd808>	 tonythomas: I'll see what I can do to help you. Joining project now.
[15:54:31] <tonythomas>	 bd808: thank you. I just created a new instance ( the error still persists ) 
[15:54:50] <bd808>	 !log newsletter Added myself to project to help debug MediaWiki-Vagrant issues
[15:54:56] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Newsletter/SAL, Master
[15:57:40] <bd808>	 tonythomas: a couple of meta notes about the steps you listed above. you shouldn't be sudo'ing as the vagrant user anymore. With the real Vagrant binary now in place you just do things as yourself.
[15:58:00] <tonythomas>	 bd808: as my localuser ?
[15:58:11] <bd808>	 yeah
[15:58:18] <tonythomas>	 oh. let me just check 
[15:58:28] <bd808>	 i'm starting the vm now
[15:58:34] <tonythomas>	 tonythomas01@newsletter-test:/srv/mediawiki-vagrant$ type -a vagrant
[15:58:34] <tonythomas>	 vagrant is /usr/bin/vagrant
[15:58:34] <bd808>	 it is running puppet
[15:58:36] <tonythomas>	 okey :) 
[15:58:59] <bd808>	 tonythomas: hmmm... did you log out and then back in after puppet allied the role?
[15:59:22] <bd808>	 vagrant should end up being an alias to /usr/local/bin/mw-vagrant
[15:59:25] <tonythomas>	 I did logout from root shell, to my user shell - and came back :\ 
[15:59:33] <tonythomas>	 looks like that was a mistake 
[16:00:05] <tonythomas>	 yeah. now - tonythomas01@newsletter-test:~$ type -a vagrant
[16:00:05] <tonythomas>	 vagrant is aliased to `/usr/local/bin/mwvagrant'
[16:00:06] <tonythomas>	 vagrant is /usr/bin/vagrant
[16:00:10] <bd808>	 ah. yeah you need to completely log out our source /etc/profile.d/alias-vagrant.sh
[16:00:21] <bd808>	 s/our/or/
[16:00:44] <tonythomas>	 bd808: okey. so 'vagrant up' is being run now ?
[16:00:54] <bd808>	 yeah
[16:01:02] <tonythomas>	 I get a 'but another process is already executing an action on the machine.... ' :D great 
[16:01:30] <tonythomas>	 in the meantime, I want to tell step 5 in https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs#Setting_up_your_instance_with_MediaWiki-Vagrant is confusing :( 
[16:01:50] <tonythomas>	 ( labs are not for normal, though ) 
[16:02:20] <bd808>	 Another bit of info is that the problem running setup.sh is known. Puppet will run it for you properly so if you hit that after updating the mediawiki-vagrant clone just force a puppet run (`sudo puppet agent --test --verbose`)
[16:03:00] <bd808>	 tonythomas: edits welcome if you can think of better wording
[16:03:14] <tonythomas>	 bd808: I will. Its the second time I get lost over there ;) 
[16:03:30] <tonythomas>	 adding it to my wishlist for editing 
[16:03:50] <bd808>	 The setup.sh problem is documented at -- https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs#Update_the_custom_MediaWiki-Vagrant_Vagrant_plugin.3F
[16:03:56] <bd808>	 that may need some help too
[16:04:20] <tonythomas>	 bd808: yeah. true
[16:04:34] <tonythomas>	 bd808: ah ! i see the apache test page in http://newsletter-test.wmflabs.org/
[16:04:38] <tonythomas>	 progress ;) 
[16:04:53] <bd808>	 puppet is still chugging along
[16:05:01] <bd808>	 inside the vm
[16:05:09] <tonythomas>	 okey 
[16:05:13] <bd808>	 which is inside the other vm :)
[16:05:20] <bd808>	 it's vms all the way down
[16:05:57] <tonythomas>	 bd808: yeah! 
[16:08:18] <tonythomas>	 bd808: yay ! 
[16:08:21] <tonythomas>	 http://newsletter-test.wmflabs.org/ shows the wiki 
[16:08:35] <tonythomas>	 let me install the newsletter extension 
[16:08:40] <tonythomas>	 I can do it from my local account, right ?
[16:08:49] <bd808>	 tonythomas: cool. `vagrant up` just finished so it's all yours from here
[16:09:16] <tonythomas>	 bd808: Thank you. someday, I should be reading 'labs for noobs' 
[16:09:43] <bd808>	 tonythomas: you should be able to, yes. If you hit permissions issues chown/chmod things to be writable by the wikidev group and try again
[16:09:58] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Create a larger newsletter-test instance in labs - https://phabricator.wikimedia.org/T120516#1955701 (10scfc) AFAICT that (T86282) is a different issue in MediaWiki-Vagrant's Puppet.  The error quoted by @01tonythomas is about the Ruby gem that is run as part of the `v...
[16:10:10] <tonythomas>	 bd808: well. I get the persmission issue :*( 
[16:10:12] <bd808>	 also we should get a mw-vagrant role for newletter setup if there isn't one already
[16:10:46] <tonythomas>	 -rw-rw-r-- 1 mwvagrant wikidev 3353 Jan 22 15:47 LocalSettings.php
[16:11:07] <bd808>	 you should be in the wikidev group
[16:14:29] <wikibugs>	 6Labs, 10Tool-Labs: Install pthreads on Tool Labs - https://phabricator.wikimedia.org/T123945#1955718 (10scfc) 5Open>3declined
[16:14:38] <tonythomas>	 bd808: just one more thing. tonythomas01@newsletter-test:/srv/mediawiki-vagrant/mediawiki/maintenance$ php update.php 
[16:14:38] <tonythomas>	 -bash: php: command not found
[16:15:17] <tonythomas>	 only mwvagrant user has it in its alias ?
[16:15:50] <tonythomas>	 neither it has :( 
[16:15:53] <bd808>	 tonythomas: yeah. remember that this is like using Vagrant on you laptop. You need to use `vagrant ssh` to get inside of the LXC container that is actually running your code
[16:16:17] <tonythomas>	 bd808: right. vm -> vm -> anothervm 
[16:17:40] <bd808>	 *nod* physical host in Labs -> newsletter-test.newsletter.eqiad.wmflabs -> Vagrant managerd LXC container
[16:18:19] <bd808>	 so do your php and mysql work inside the LXC container by opening a session to it with `vagrant ssh`
[16:18:20] <tonythomas>	 bd808: and its loaded from /var/www/w right ?
[16:19:34] <bd808>	 yes. /var/www/w is where the MultiVersion stubs live
[16:19:51] <bd808>	  /var/www is the docroot for apacha
[16:20:03] <bd808>	 *apache
[16:20:06] <tonythomas>	 and I dont see a maintenance folder and the update.php :D my bad 
[16:20:26] <bd808>	 $IP is /vagrant/mediawiki
[16:20:40] <bd808>	 just like on your laptop
[16:20:49] <bd808>	 (assuming you use mediawiki-vagrant)
[16:21:19] <tonythomas>	 bd808: I have mediaiwki-vagrant configured, but I have it hosted locally in my laptop :D 
[16:23:10] <tonythomas>	 bd808: great. http://newsletter-test.wmflabs.org/wiki/Special:CreateNewsletter now works 
[16:23:11] <tonythomas>	 THank you 
[16:23:16] <wikibugs>	 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1204 puppet tries to downgrade php-* packages - https://phabricator.wikimedia.org/T123836#1955760 (10scfc) a:3scfc This error happens when packages are installed by Puppet before the WMF repository is set up, and the package in the WMF repository is older than the o...
[16:25:16] <bd808>	 tonythomas: you are welcome. Pay me back by trying to make the docs better :)
[16:25:42] <tonythomas>	 bd808: I will :) 
[16:29:28] <wikibugs>	 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1204 puppet tries to downgrade php-* packages - https://phabricator.wikimedia.org/T123836#1955768 (10scfc) 5Open>3Resolved Purged the Ubuntu packages, ran Puppet twice:  ``` scfc@tools-webgrid-lighttpd-1204:~$ sudo puppet agent -tv Info: Retrieving plugin Info: Lo...
[16:38:06] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1955803 (10Johan) >>! In T120497#1944237, @NiharikaKohli wrote: >>>! In T120497#1944221, @Milimetric wrote: >> My opinion: >>  >> We should bring together the go...
[16:49:36] <freephile>	 If I want to contribute a one-line change to the Lingo extension, what's the best way to do that?  In github it would just be a fork and pull request.  I have a wikitech and a gerrit account, but currently unable to ssh to bastion.
[16:52:21] <valhallasw`cloud>	 freephile: git diff + https://github.com/valhallasw/gerrit-patch-uploader
[16:52:30] <valhallasw`cloud>	 eh, https://tools.wmflabs.org/gerrit-patch-uploader/
[16:53:16] <valhallasw`cloud>	 the alternative is to set up git-review, which is documented here: https://www.mediawiki.org/wiki/Gerrit/Tutorial
[17:06:15] <wikibugs>	 6Labs, 10MediaWiki-extensions-Newsletter: Create a larger newsletter-test instance in labs - https://phabricator.wikimedia.org/T120516#1955882 (10bd808) >>! In T120516#1955701, @scfc wrote: > AFAICT that (T86282) is a different issue in MediaWiki-Vagrant's Puppet.  The error quoted by @01tonythomas is about th...
[17:06:27] <freephile>	 valhallasw`cloud:thanks
[17:31:55] <freephile>	 no love from the gerrit-patch-uploader... it keeps complaining about 'can't find file to patch at input line...' 
[17:32:09] <freephile>	 I've tried using plain 'git diff'
[17:32:45] <valhallasw`cloud>	 freephile: that's weird. can you post the patch in a pastebin?
[17:32:45] <freephile>	 and also creating a local bugfix branch, and git format-patch
[17:35:49] <freephile>	 http://pastebin.com/7JU0U2bP
[18:02:47] <valhallasw`cloud>	 freephile: sorry, was having dinner. Will take a look now.
[18:08:12] <valhallasw`cloud>	 freephile: error: patch failed: Lingo.php:96 error: Lingo.php: patch does not apply
[18:08:25] <valhallasw`cloud>	 freephile: so you probably need to git pull first, merge, then try again
[18:09:04] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1956127 (10Nuria) >I talked to @Ainali about this, and he mentioned that he can't really kick six persons from the project, and said it's eight or no one. I aske...
[18:13:25] <Alphos>	 hello ! as suggested by addshore in #wikidata, i'm attempting to write a tool to find entities (1) with wikilinks that redirect to pages that have their wikilinks in other entities ; this prevents editing the entities (1) from being edited in some cases. as suggested by addshore again, i'd like to use the labs db replica.
[18:14:31] <Alphos>	 am i right to understand that, now that i have created an account on wikitech, i need to request access to the tools project through https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request ?
[18:15:19] <valhallasw`cloud>	 Alphos: yep
[18:15:59] <valhallasw`cloud>	 Alphos: or you can use quarry.wmflabs.org, depending on whether it can be done in pure sql
[18:16:02] <Alphos>	 i asked for a shell account when registering, so do i need to re-request one through https://wikitech.wikimedia.org/wiki/Special:FormEdit/Shell_Access_Request ?
[18:16:27] <Alphos>	 valhallasw`cloud it can't be done in pure sql, wikidata and sql aren't really in the same plane ;)
[18:16:40] <valhallasw`cloud>	 Alphos: fair enough :-)
[18:16:49] <valhallasw`cloud>	 no, the shell request is not necessary anymore
[18:16:54] <Alphos>	 damn, joining would have been so good :D
[18:16:56] <Alphos>	 ok thanks :)
[18:47:40] <Alphos>	 out of curiosity, is is safe to expect an answer to my request on monday ? this way i have a reason for not doing a thing this weekend ^^
[18:56:56] * darkblue_b reads "Community Consultation Process" email with interest
[18:57:38] <valhallasw`cloud>	 Alphos: well, we can wait until monday or I can click the button now ;-)
[18:57:47] <valhallasw`cloud>	 I was expecting a bot to report the change, but apparently not.
[18:58:59] <valhallasw`cloud>	 "Successfully added Alphos to tools."
[18:59:29] <valhallasw`cloud>	 Alphos: but feel free to not do anything this weekend anyway :-)
[19:00:17] <Alphos>	 darn :-(
[19:00:24] <Alphos>	 now i have no other choice :-(
[19:07:20] <Alphos>	 as an old apache friend of mine once said, "It works!" :) thanks again ^^
[19:08:10] <valhallasw`cloud>	 :-)
[19:09:29] <valhallasw`cloud>	 you're welcome. The pages linked on your talk page have a lot of background info on how tool labs is setup, as it's a bit different from a typical shared hosting setup. Please let us know if you have any issues! :-)
[19:30:01] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: GridEngine down due to bdb issues - https://phabricator.wikimedia.org/T122638#1956642 (10Aklapper) https://gerrit.wikimedia.org/r/264390 got merged a week ago - has this issue been seen again? Is this task resolved? Or is more work needed?
[19:33:39] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10DBA, 6operations: db1069 is running low on space - https://phabricator.wikimedia.org/T124464#1956656 (10jcrespo) 3NEW a:3jcrespo
[19:39:43] <Alphos>	 valhallasw`cloud was already browsing through them ^^ as it turns out, i'll have to phabricate for a replica.my.cnf file ^^'
[19:40:19] <valhallasw`cloud>	 Alphos: this is our sneaky way to make sure everyone has a phabricator account! (just kidding)
[19:40:29] <Alphos>	 teehee :d
[19:40:30] <Alphos>	 :D
[19:40:49] <valhallasw`cloud>	 unfortunately, I can't help you with that as it requires access to a server I don't have access to, but hopefully those that do will have time to look at it
[19:41:43] <Alphos>	 no problem, i'm not in any hurry. checking on phab if there's a pending issue with creating them for other users
[19:47:03] <Alphos>	 apparently not - well, not one for tools anyway, just one user for his user account
[19:49:47] <valhallasw`cloud>	 Alphos: yeah, it's supposed to Just Work(TM), although it could be it's a task that only runs once an hour or so (not entirely sure)
[19:50:20] <Alphos>	 didn't think of that. I'll give it a few more minutes before hitting the "Create task" button :)
[19:53:52] <valhallasw`cloud>	 Alphos: hm, I think it should be ~120s between runs, not an hour
[19:54:21] <valhallasw`cloud>	 so I think creating a task might still be a good idea
[19:54:25] <Alphos>	 bah, waiting till 21:05 CET won't really affect me much
[19:54:51] <Alphos>	 10 minutes isn't really a long time, i won't scream because i didn't get my cookies ^^
[19:55:08] <Alphos>	 won't be writing a single useful LoC til monday evening anyway
[20:03:08] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: GridEngine down due to bdb issues - https://phabricator.wikimedia.org/T122638#1956815 (10valhallasw) It seems it's still happening: ``` 01/16/2016 12:00:27|worker|tools-grid-master|E|error writing object with key "USER:tools.sbot" into berkeley database: (22) Invalid ar...
[20:04:28] <grrrit-wm>	 (03PS1) 10MtDu: Add PLURAL support for SHOW_LAST_DAYS message [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265804 
[20:14:25] <wikibugs>	 6Labs, 10Tool-Labs: Possible race condition in webservice HSET/HDEL - https://phabricator.wikimedia.org/T122515#1956861 (10valhallasw) I can confirm this is what is happening. When I try to reschedule crosswatch, the following happens:  ``` valhallasw@tools-proxy-01:~$ tail -f /var/lib/redis/tools-proxy-01-637...
[20:15:30] <wikibugs>	 6Labs, 10Tool-Labs: Missing replica.my.cnf for wikidata-redirects-conflicts-reports - https://phabricator.wikimedia.org/T124470#1956866 (10Alphos) 3NEW
[20:29:00] <ottomata>	 it is possible i have not been following some information
[20:29:05] <ottomata>	 but, i see that I cannot create a jessie instance?
[20:29:27] <ottomata>	 andrewbogott: ^
[20:29:34] <ottomata>	 is this related to kernel exploit?
[20:31:18] <valhallasw`cloud>	 ottomata: yes, jessie instance creation is broken
[20:31:33] <ottomata>	 k danke
[20:36:21] <valhallasw`cloud>	 ottomata: as I understand it, a new base image has to be built with the new kernel, and that hasn't happened yet. 
[20:36:59] <ottomata>	 k thanks
[20:50:12] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Added dummy password for passwords::openldap::labtest [labs/private] - 10https://gerrit.wikimedia.org/r/265827 
[20:51:20] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032 V: 032] Added dummy password for passwords::openldap::labtest [labs/private] - 10https://gerrit.wikimedia.org/r/265827 (owner: 10Andrew Bogott)
[21:48:39] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1957269 (10ezachte) If we have more human resources than functional requirements, I'd like to propose this idea: what about making this new UI language independe...
[21:53:44] <chippy>	 I think there may be much slower read times from the home partition labstore.svc.eqiad.wmnet:/project/maps/home  to the instance maps-warper It seems noticeably slower but I'm unable to run hdparm on that network fs
[21:53:56] <wikibugs>	 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#1957305 (10Ainali) >>! In T120497#1957269, @ezachte wrote: > If we have more human resources than functional requirements, I'd like to propose this idea: what ab...
[21:55:31] <chippy>	 would there be any way to determine the speed of reads from that eqiad FS? time a copy of a 1 gig file perhaps?
[22:06:06] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 323 bytes in 0.002 second response time
[22:09:04] <andrewbogott>	 I saw ^ happening but it is already back
[22:11:08] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 983979 bytes in 2.666 second response time
[22:15:26] <valhallasw`cloud>	 andrewbogott: ldap? there's a whole bunch of unable to resolve host tools-exec-1206-like diamond mails in my inbox
[22:15:50] <andrewbogott>	 Yeah, I’ve been seeing a few ldap issues but I can’t pin anything down
[22:15:56] <andrewbogott>	 naturally it always works fine when I look right at it
[22:16:25] <valhallasw`cloud>	 andrewbogott: the timing corresponds to cron not firing on tools-submit
[22:16:30] <andrewbogott>	 Well, ok — backing up, I was messing with dns earlier.  So that is /probably/ the issue
[22:16:44] <andrewbogott>	 although the reason I was messing with dns is that designate could not contact ldap
[22:17:17] <valhallasw`cloud>	 wait, no, it doesn't line up with cron, because that was at 2300UTC, not 2300CET
[22:17:29] <valhallasw`cloud>	 (and 2400UTC)
[22:19:09] <andrewbogott>	 (I have to step away, sorry… back soon)
[22:20:53] <freephile>	 valhallasw`cloud: I setup Gerrit-review and it was merged to master https://gerrit.wikimedia.org/r/#/c/265800/
[22:22:16] <freephile>	 There was a failed test, and I'm not sure what that means.  Hopefully nothing because I don't have time to investigate :-(
[22:28:06] <andrewbogott>	 valhallasw`cloud: ok, sorry.  Here’s the whole story:
[22:28:17] <andrewbogott>	 - I was hacking on designate, so restarted designate services several times
[22:28:25] <andrewbogott>	 - A one point, designate was failing because it couldn’t contact ldap
[22:28:35] <andrewbogott>	 - After a few minutes, it started working again without my intervention
[22:28:41] <andrewbogott>	 - At no point did I restart pdns
[22:29:14] <andrewbogott>	 Note, though, that internal dns doesn’t use ldap.  So it might have been an ldap flakeout but the resolution failure is something more complicated :(
[22:32:24] <valhallasw`cloud>	 andrewbogott: ah, I somehow thought it was more closely related (nscld is related to nameserver lookups, I think?)
[22:32:30] <valhallasw`cloud>	 thanks for the clarification
[22:34:11] <andrewbogott>	 valhallasw`cloud: it is temporarily messier than it should be.  Private dns is managed by designate + powerdns (backed with a mysql database)
[22:34:16] <valhallasw`cloud>	 freephile: not sure. The output says "20:30:12 File '/mnt/jenkins-workspace/workspace/mwext-testextension-php53/src/extensions/Lingo/styles/#default#VML' referenced by 'ext.jquery.qtip' must exist." but I don't see how that's related to your change
[22:34:21] <andrewbogott>	 public dns is managed by OpenStackMAnager + a different powerdns (backed by ldap)
[22:34:37] <andrewbogott>	 but of course user accounts all come from ldap, so anything to do with logins hits ldap for sure
[22:35:10] <valhallasw`cloud>	 the internal labs DNS is part of the public dns?
[22:35:36] <valhallasw`cloud>	 rephrased more clearly: private = prod? or labs-internal?
[22:36:04] <andrewbogott>	 heh, wmf production dns is yet another system
[22:36:21] <andrewbogott>	 I was speaking only about labs.  So, private is names like foo.testlabs.eqiad.wmflabs and public is names like tools.wmflabs.org
[22:36:28] <valhallasw`cloud>	 ah, ok
[22:36:40] <andrewbogott>	 osm/pdns/ldap only manages the .wmflabs.org domain
[22:37:20] <andrewbogott>	 production dns is quite a bit less dynamic — there are just static files in git for that
[22:37:41] <valhallasw`cloud>	 ah, of course. Yes, I have submitted patches for those at some point in the past
[22:38:39] <valhallasw`cloud>	 andrewbogott: again, thanks for the explanation :-)
[22:39:05] <andrewbogott>	 valhallasw`cloud: part of the quarterly goal this quarter is to kill off the osm/pdns/ldap track
[22:39:12] <andrewbogott>	 at which point things will be ever so slightly simpler
[23:04:13] <subbu>	 i am trying to create a labs instance for building some node modules to use on ruthenium .. but looks like I don't have permissions to create a new instance.
[23:04:35] <subbu>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep lists me as an admin.
[23:07:04] <subbu>	 andrewbogott, YuviPanda chasemp what is a good way of doing this?
[23:07:39] <andrewbogott>	 subbu: just a second...
[23:08:11] <mutante>	 i can handle this. hold on
[23:08:23] <andrewbogott>	 subbu: why do you think you don’t have permissions?
[23:09:20] <subbu>	 i am looking at https://wikitech.wikimedia.org/wiki/Help:Instances and i don't see " In the left menu look for the sub-menu group “Labs Projectadmins” (this will only be visible after logging in to wikitech) and click on it to open it."
[23:09:44] <andrewbogott>	 ‘Manage instances’ is what you want
[23:10:00] <subbu>	 i don't see such a link in the left menu.
[23:10:01] <mutante>	 subbu: what's your username thre
[23:10:16] <subbu>	 'Subramanya Sastry'
[23:11:08] <mutante>	 subbu: try switching to a new project:
[23:11:10] <mutante>	 called "nodejs"
[23:11:23] <mutante>	 this is better than using deployment-prep anyways
[23:11:28] <mutante>	 and i made you admin
[23:11:54] <mutante>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Nodejs
[23:12:01] <mutante>	 yes, you are not listed on that page
[23:12:06] <subbu>	 ok. i see all those links now.
[23:12:07] <subbu>	 thanks.
[23:12:08] <mutante>	 but i think that's because there are other issues
[23:12:16] <mutante>	 cool
[23:12:19] <andrewbogott>	 eh, the nova_resource pages only update periodically
[23:12:25] <andrewbogott>	 so if you only just changed something it wouldn’t take effect
[23:12:26] <mutante>	 so i think the Nove_Resource pages are not up2date
[23:12:32] <mutante>	 that
[23:12:33] <mutante>	 ok
[23:13:28] <mutante>	 andrewbogott: can i delete a project via web ui?
[23:13:37] <subbu>	 mutante, hmm .. there is no jessie instance type though .. only ubuntu ... 
[23:13:38] <andrewbogott>	 mutante: I don’t know if you can, but I can
[23:14:20] <mutante>	 andrewbogott: if you could delete "node" , but keep "nodejs". i did both by accident 
[23:14:28] <andrewbogott>	 mutante: ok
[23:14:31] <mutante>	 thanks
[23:15:05] <mutante>	 hmm.. i have jessie on my instance though..
[23:15:10] <andrewbogott>	 mutante: it’s empty, I trust?
[23:15:18] <mutante>	 andrewbogott: yes, just created, empty
[23:15:20] <subbu>	 i wonder if i build the node modules on the ubuntu 14.04 and try it on ruthenium which is jessie ... if it would work.
[23:15:40] <mutante>	 i really thought we have a jessie image.. eh...
[23:15:57] <mutante>	 didnt i already use that
[23:19:11] <valhallasw`cloud>	 jessie instance creation is broken at the moment
[23:19:44] <mutante>	 thanks
[23:26:23] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Tool-Labs: Backports are enabled in new Trusty instances - https://phabricator.wikimedia.org/T123890#1957860 (10Andrew) I agree that the ways of cloud-init are mysterious.  I dug in the source a bit but don't see where the default behavior is described... The current default b...
[23:28:13] <wikibugs>	 6Labs, 6Stewards-and-global-tools: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1957871 (10Andrew) Sorry, I can't tell what this is about :)  The project will be used to host a test wiki?
[23:30:01] <chasemp>	 YuviPanda: andrewbogott I have a setting on labstore1001 staged with puppet disable and I'm loath to undo it as it seems to be doing well but I have to step away for a bit
[23:30:18] <chasemp>	 I'll shoot an email quick with details but does this bother you in general?
[23:30:49] <andrewbogott>	 chasemp: it’s ok with me — you left a ‘disabled’ message explaining, and turned off the alert on icinga?
[23:30:58] <chasemp>	 I did
[23:30:59] <wikibugs>	 6Labs, 6Stewards-and-global-tools: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1957880 (10Dzahn) it's about tools used by these people  https://en.wikipedia.org/wiki/Wikipedia:Functionaries
[23:32:16] <wikibugs>	 6Labs, 6Stewards-and-global-tools: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1957887 (10Andrew) OK, that makes sense.  'work on tools' is still pretty vague, though -- my question about wiki hosting is because that's an example of something that can't easi...
[23:56:33] <wikibugs>	 6Labs, 6Stewards-and-global-tools: Create a Labs project for admin tooling enhancements - https://phabricator.wikimedia.org/T123993#1958036 (10yuvipanda) I'm pretty sure this is going to be hosting testwikis to test improvements to MW.   I'm hoping we can avoid calling it anythingtools to avoid some confusion :D