[00:06:00] <legoktm>	 YuviPanda: wanna figure out why grrrit-wm is down?
[00:06:34] <greg-g>	 YuviPanda: btw, welcome to SF
[00:06:37] <YuviPanda>	 legoktm: I think I know already. Only one exec node is trusty and it is massively overloaded.
[00:07:08] <legoktm>	 can you create another exec node? :D
[00:07:14] <YuviPanda>	 legoktm: I created 5 new trusty nodes, they are provisioning now. However I dont have my laptop so I can't actually add them :(
[00:07:19] <legoktm>	 :|
[00:07:26] <YuviPanda>	 greg-g: wheeeee :)
[00:07:53] <YuviPanda>	 legoktm: been out for a while now. Need to poke someone else with root to do the actual addition 
[00:08:29] <legoktm>	 valha...oh wait :(
[00:09:15] <YuviPanda>	 legoktm: valhallanotyet
[00:11:10] <YuviPanda>	 greg-g: still quite groggy tho. Happy I picked Friday to land and not monday
[00:11:15] <greg-g>	 no kidding
[01:44:02] <Negative24>	 twentyafterfour: I didn't realize that master is farther behind than production (this rolling release really makes git history look bad for us)
[02:08:48] <twentyafterfour>	 Negative24: that's not upstream master, that's our fork of it
[02:08:57] <Negative24>	 exactly
[02:09:25] <Negative24>	 I thought though that master would be preserved as the latest cherry-picked upstream commits
[02:10:13] <Negative24>	 So have you seen anything?
[02:10:25] <twentyafterfour>	 anything?
[02:10:41] <Negative24>	 well you've been logged into phab-02 for quite some time
[02:11:12] <Negative24>	 I've been replaying git commits from the time the problem was recognized by me back
[02:11:30] <twentyafterfour>	 the top nav was broken when I was logged out but I haven't seen any further problems since logging in
[02:12:10] <Negative24>	 the breadcrumb nav?
[02:14:40] <twentyafterfour>	 yeah
[02:14:52] <twentyafterfour>	 actually you know what the difference is - chrome
[02:14:58] <twentyafterfour>	 firefox is fine, chrome is broken
[02:15:40] <Negative24>	 twentyafterfour: this is what I'm seeing http://i.imgur.com/keG8fMz.png
[02:16:04] <twentyafterfour>	 yep ... now try firefox
[02:16:12] <Negative24>	 firefox is completely fine?
[02:16:30] <twentyafterfour>	 seems like it
[02:17:21] <Negative24>	 logged in and out?
[02:17:35] <twentyafterfour>	 yep
[02:17:41] <Negative24>	 hrm
[02:18:04] <Negative24>	 Well that's upstream then. I'm going to search their sources for a bug report
[02:18:19] <Negative24>	 I disabled extensions for a short time and it didn't fix it
[02:21:57] <twentyafterfour>	 https://phab-02.wmflabs.org/file/data/jdd7tzdlx2scfmceinml/PHID-FILE-bju3epybeeky2iiaj5nn/phab_screenshot
[02:24:20] <Negative24>	 I'm not reproducing on firefox
[02:24:37] <Negative24>	 still shows up the same way logged in and out
[02:27:37] <twentyafterfour>	 this is what I see in chrome:  https://phab-02.wmflabs.org/file/data/abtfzxhifljk5gl42qtr/PHID-FILE-3e45x5e3ishxlvagqln3/phab_in_chrome
[02:29:18] <Negative24>	 this is firefox for me: https://phab-02.wmflabs.org/file/data/deikjst3lipki4paibl6/PHID-FILE-nb5lncqk6byotpkhr3ei/Phab-02_in_firefox
[02:33:08] <twentyafterfour>	 that's really strange
[02:34:11] <Negative24>	 yup
[02:35:14] <Negative24>	 maybe one of our firefox's are out of date
[02:36:56] <Negative24>	 mines at 36.0.4 (36.0.4+build1-0ubuntu0.14.04.1)
[02:37:34] <twentyafterfour>	 36.0.4
[02:37:55] * Negative24 sighs
[02:40:26] <Negative24>	 I'll submit a bug report upstream and see if they can figure it out
[02:41:40] <Negative24>	 Funny thing was is that it happened before and then it resolved it's self
[02:42:21] <twentyafterfour>	 ok here's the difference between firefox and chrome:  they are getting served a different version of the css:     chrome is broken, with this css: https://phab-02.wmflabs.org/res/phabricator/f1eab25d/core.pkg.css
[02:42:47] <twentyafterfour>	 firefox is working and it's getting this css file: https://phab-02.wmflabs.org/res/phabricator/65e04767/core.pkg.js
[02:43:54] <twentyafterfour>	 er
[02:44:03] <twentyafterfour>	 sorry wrong paste
[02:45:51] <twentyafterfour>	 bah I'm making shit up
[02:46:36] <Negative24>	 ?
[02:46:45] <twentyafterfour>	 false alarm
[02:46:48] <Negative24>	 those css files seem to be generated
[02:46:55] <twentyafterfour>	 I thought I had something there but I was reading wrong
[02:47:07] <Negative24>	 I bet it is css though
[02:47:32] <twentyafterfour>	 yeah if you look at it in the DOM inspector the css for that panel is all missing
[02:48:19] <Negative24>	 why are all the setup configuration issues gone now?
[02:48:48] <Negative24>	 oh you dismissed them
[02:51:25] <Negative24>	 I *really* wish I knew what triggered this
[03:21:53] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Vihari was created, changed by Vihari link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fVihari edit summary: Created page with "{{Tools Access Request |Justification=To create new tools |Completed=false |User Name=Vihari }}"
[03:50:48] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Vihari was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=150645 edit summary: 
[03:51:03] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Suriyaa Kudo was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=150647 edit summary: 
[04:31:53] <PhantomTech>	 https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Submitting_continuous_jobs_.28such_as_bots.29_with_.27jstart.27 says a job started with jstart will automaticaly be restarted but right below that it explains how to setup bigbrother to restart a jstart job for you
[04:48:47] <L235>	 Hi. When I put http://tools.wmflabs.org/badges/manual/v1/test1.json into http://validator.openbadges.org/ I get a 403 error
[04:49:01] <L235>	 (works with everything off of tool labs)
[04:49:38] <Krenair>	 might be better to ask the operators or openbadges.org
[04:49:42] <Krenair>	 operators of*
[04:49:50] <L235>	 Krenair: it works for everything not on Tool Labs
[04:49:58] <Krenair>	 ok
[04:50:11] <Krenair>	 so ask them what is wrong with tools' response
[04:50:34] * L235 had a feeling that would be the response
[04:51:18] <Krenair>	 It returns 200 to me.
[04:51:40] <L235>	 Krenair: I've done a lot of testing
[04:51:45] <Krenair>	 (The file from tools, not openbadges.org)
[04:52:29] <L235>	 yes, http://tools.wmflabs.org/badges/manual/v1/test1.json returns 200 for everyone except openbadges, and everything else works for openbadges
[04:52:56] <L235>	 (this is a problem on Mozilla Backpack as well)
[04:55:04] <L235>	 I have a feeling I annoyed everything too much
[04:55:09] <L235>	 everyone*
[04:55:47] <wctaiwan>	 Not that I'm aware of.
[05:01:52] * L235 taps Krenair- any thoughts?
[05:02:17] <legoktm>	 L235: does your access log show the 403s?
[05:02:42] <L235>	 don't think so
[05:02:57] <legoktm>	 umm
[05:02:58] <legoktm>	 check?
[05:03:27] <L235>	 wait, maybe
[05:03:35] <Krenair>	 maybe?
[05:04:00] <L235>	 no, they aren't
[05:04:14] <Krenair>	 they aren't 403s?
[05:04:33] <L235>	 I don't see any 403s in the access log
[05:04:41] <legoktm>	 if it's not in the log, something in the proxy is preventing openbadges.org from reaching your tool, you should probably file a bug about it
[05:05:58] <L235>	 that'll go in the Tool-labs project?
[05:06:18] <L235>	 (the phabricator bug, that is)
[05:07:27] <wctaiwan>	 yes
[05:33:17] <wikibugs>	 10Tool-Labs: openbadges.org not connecting to tool labs - https://phabricator.wikimedia.org/T94332#1159979 (10Legoktm)
[05:33:49] <legoktm>	 L235: you actually have to add the project ^ :P
[05:34:25] <L235>	 oh, I thought I did... thanks ;)
[06:35:21] <SweetyDumling>	 legoktm for over 36 hours now...:
[06:35:23] <SweetyDumling>	 "No webservice
[06:35:24] <SweetyDumling>	 The URI you have requested, /xtools-articleinfo/index.php?article=Germanwings_Flight_9525&lang=en&wiki=wikipedia, is not currently serviced."
[06:38:34] <legoktm>	 SweetyDumling: the maintainer of xtools-articleinfo has to fix it
[06:38:58] <SweetyDumling>	 and where is that tool, physically?
[06:40:20] <legoktm>	 what do you mean by physically?
[06:40:51] <SweetyDumling>	 where is the server that has it installed
[06:40:59] <legoktm>	 it's in virginia
[06:41:14] <SweetyDumling>	 but the maintainer could be anywhere?
[06:41:21] <legoktm>	 yup
[06:46:24] <SweetyDumling>	 legoktm left T13 a talk message, as that person is most personable ;) of the listed three maintainers.  Someoen already did on 27 march, so it has been a while.
[06:48:02] <legoktm>	 sounds good
[06:54:25] <SweetyDumling>	 971 linked-in connections.  shooting for 1001, as in the # of nights in Tales of the Arabian Nights.  https://linkedin.com/in/mareklug
[07:06:58] <gifti>	 my error.log shows:
[07:06:59] <gifti>	 2015-03-28 21:42:46: (server.c.1512) server stopped by UID = 0 PID = 25887
[07:07:02] <gifti>	 2015-03-28 21:43:07: (network.c.358) can't bind to port:  14005 Address already in use
[07:07:06] <gifti>	 2015-03-29 07:04:52: (network.c.358) can't bind to port:  14005 Address already in use
[07:07:12] <gifti>	 last line was a manual start attempt
[07:07:28] <gifti>	 what's going wrong exactly and who wants to fix it this time?
[07:07:43] <gifti>	 (tools.giftbot)
[07:08:04] <gifti>	 could i fix it myself?
[07:13:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:14:21] <gifti>	 YuviPanda, Coren, (who else?) ↑
[11:34:38] <GerardM->	 https://tools.wmflabs.org/magnustools/multistatus.html can somebody please start CatScan2
[11:44:13] <valhallasw>	 !log tools.lolrrit-wm grrrit-wm still looks broken. Trying to run manually on tools-bastion-02
[11:44:16] <labs-morebots>	 Logged the message, Master
[11:45:24] * valhallasw prods grrrit-wm 
[11:48:10] <gifti>	 hm, problem seems to have solved itself
[11:49:36] <valhallasw>	 !log tools.lolrrit-wm *maybe* the issue is it can't join #wmt, although it's not clear to me why that should silently kill all output
[11:49:39] <labs-morebots>	 Logged the message, Master
[11:51:56] <valhallasw>	 !log tools.lolrrit-wm I also don't see any mention of connecting to gerrit.wikimedia.org in the log file. Last message was yesterday, 2015-03-28T10:57:47.422Z - info: Sent message from extensions/EducationProgram to #mediawiki-feed
[11:51:58] <labs-morebots>	 Logged the message, Master
[11:51:59] <GerardM->	 there are issues with WDQ 
[11:52:29] <GerardM->	 ir also affects Reasonator https://tools.wmflabs.org/reasonator/?&q=6590784
[11:53:38] <valhallasw>	 YuviPanda, can you take a look at grrrit-wm? I can't figure out what's wrong.
[11:55:26] <valhallasw>	 YuviPanda: added some more logging info, maybe that helps...
[11:57:09] <valhallasw>	 !log tools.lolrrit-wm Added logging to log connecting to gerrit, but it doesn't seem to do that at all. What the heck?!
[11:57:12] <labs-morebots>	 Logged the message, Master
[12:00:23] <GerardM->	 ... the environment is not usable at this time for me
[12:00:55] <GerardM->	 I have some processes running ... they go on, I cannot start anything at this time
[12:02:14] <valhallasw>	 !log tools.lolrrit-wm YES! Ok, connected to event stream now. The issue was A) not being able to join #wmt, which means the number of channels joined was wrong, and B) checking for config.nick, so testing with a second grrrit-wm didn't work
[12:02:17] <labs-morebots>	 Logged the message, Master
[12:02:43] <grrrit-wm1>	 (03CR) 10Merlijn van Deen: [C: 04-1] "derp" [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/185644 (owner: 10Merlijn van Deen)
[12:02:51] <valhallasw>	 jahoorhijdoethetweer
[12:04:43] <GerardM->	 valhallasw: Thanks, het lijkterop
[12:05:02] <valhallasw>	 I haven't done anything for WDQ (also don't have the access to do that)
[12:08:12] <valhallasw>	 !log tools.lolrrit-wm See https://gerrit.wikimedia.org/r/200351 ; cleared log file and rescheduled job. Hopefully it'll work now :/
[12:08:15] <labs-morebots>	 Logged the message, Master
[12:09:57] <valhallasw>	 !log tools.lolrrit-wm "2015-03-29T12:09:30.093Z - info: Connected to event stream!"
[12:10:00] <labs-morebots>	 Logged the message, Master
[12:17:46] <grrrit-wm>	 (03Abandoned) 10Merlijn van Deen: hacky script to dump mysql subscriptions to redis [labs/tools/gerrit-to-redis] - 10https://gerrit.wikimedia.org/r/185644 (owner: 10Merlijn van Deen)
[12:51:47] <GerardM->	 NB Reasonator is still down
[12:52:42] <GerardM->	 HTML is not being served ? https://tools.wmflabs.org/magnustools/multistatus.html also does not show stuff
[12:55:18] <wikibugs>	 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1160191 (10valhallasw) 5Resolved>3Open
[12:58:41] <valhallasw>	 GerardM-: ^
[16:50:46] <oleg3280>	 please correct work of the service https://tools.wmflabs.org/guc/?user=Oleg3280 (extra slash for Russian projects of the Wikimedia Foundation) (http:/// instead of http://) (links do not work) (for comparison https://tools.wmflabs.org/pirsquared/guc/?user=Oleg3280 working correctly). thanks.
[16:53:56] <valhallasw>	 oleg3280: please file a bug under https://phabricator.wikimedia.org/maniphest/task/create/?projects=Tool-Labs-tools-Global-user-contributions
[16:54:30] <valhallasw>	 oleg3280: and cc user:luxo
[17:06:10] <wikibugs>	 10Tool-Labs-tools-Global-user-contributions: https://tools.wmflabs.org/guc - https://phabricator.wikimedia.org/T94351#1160352 (10Oleg3280) 3NEW
[17:17:03] <wikibugs>	 10Tool-Labs-tools-Global-user-contributions: https://tools.wmflabs.org/guc - https://phabricator.wikimedia.org/T94351#1160379 (10Oleg3280)
[17:20:46] <afeder>	 For an application running on the web grid, is it good practice to re-dispatch non-trivial processing to the standard grid queue? If so, what is the rough threshold for 'non-triviality'?
[18:43:06] <GerardM->	 any update on the problems with Reasonator et al ?
[18:43:32] <GerardM->	 https://tools.wmflabs.org/magnustools/multistatus.html this does not show any content either
[18:59:06] <wikibugs>	 6Labs, 10Tool-Labs, 7Tracking: Provide 'Support request' tool labs project - https://phabricator.wikimedia.org/T94359#1160502 (10valhallasw) 3NEW
[18:59:43] <wikibugs>	 6Labs, 10Tool-Labs: Provide 'Support request' tool labs project - https://phabricator.wikimedia.org/T94359#1160502 (10valhallasw)
[19:11:25] <wikibugs>	 6Labs: Sync up the new labs NFS project filesystem with the live one - https://phabricator.wikimedia.org/T93792#1160528 (10Technical13) Bump... What is the status on this?  I'm assuming this is the reason that when I SSH into labs I'm 'technical-13@labs-bastion-01~' instead of 'technical-13@tools-login' (or what...
[19:26:32] <wikibugs>	 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1160542 (10scfc) There is no new entry in the tool's `bigbrother.log`, but no web service either.  `tools-submit` was rebooted six days ago, and maybe it caused the same condition with `bigbrot...
[19:26:51] <wikibugs>	 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1160543 (10scfc) There is no new entry in the tool's `bigbrother.log`, but no web service either.  `tools-submit` was rebooted six days ago, and maybe it caused the same condition with `bigbrot...
[19:28:24] <valhallasw>	 GerardM-: ^
[19:29:49] <wikibugs>	 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1160546 (10scfc) 5Open>3Resolved Actually, remembering caused me to run `jsub sleep 30`, which was enough to make `bigbrother` aware of `magnustools` again, and it has started the web servi...
[19:30:35] <GerardM->	 that made a difference.. It lost me more than a days of work
[19:32:38] <wikibugs>	 6Labs, 10Tool-Labs: Provide 'Support request' tool labs project - https://phabricator.wikimedia.org/T94359#1160550 (10Krenair) urgh, you want to store support requests alongside actual tasks?
[19:37:16] <wikibugs>	 6Labs, 10Tool-Labs: Provide 'Support request' tool labs project - https://phabricator.wikimedia.org/T94359#1160556 (10valhallasw) Yes and no. I'd like to have them in phabricator, but not mixed in with the actual to-do list (hence a new project). It's much better to have them here, as phabricator is searchable...
[19:39:35] <GerardM->	 https://tools.wmflabs.org/magnustools/multistatus.html catscan2 is down
[19:41:51] <gifti>	 catscan3 should be working
[19:47:39] <valhallasw>	 GerardM-: file a bug please
[19:49:40] <wikibugs>	 6Labs, 10Tool-Labs: Provide 'Support request' tool labs project - https://phabricator.wikimedia.org/T94359#1160566 (10scfc) I like the idea of directing all issues to Phabricator not least because it provides a record that others can revisit to see how similar issues have been resolved in the past.  I'm not so...
[19:53:48] <wikibugs>	 6Labs: Sync up the new labs NFS project filesystem with the live one - https://phabricator.wikimedia.org/T93792#1160584 (10scfc) No, the "change" in the host name (actually, you are connecting to a different host now) was due to adding new bastion hosts for the Tools project; cf. @yuvipanda's mail at http://perm...
[20:16:45] <GerardM->	 valhallasw: I have no time for that
[20:16:48] <GerardM->	 literally
[21:12:17] <YuviPanda>	 GerardM-: to repeat being more respectful is going to help get you help faster. Thank you. 
[21:13:06] * YuviPanda adds a comma after 'to repeat'
[21:13:09] <GerardM->	 I am sorry but I do not have time at this time
[21:13:51] <YuviPanda>	 GerardM-: then you should not expect other people to make time for things on weekends either. You are coming off as demanding and boorish and that is not going to help you. 
[21:13:59] * YuviPanda goes back to his weekend
[23:19:21] <Negative24>	 Is anyone acquainted with nginx's proxy setup?
[23:20:42] <Krenair>	 Well, someone set it up. So I'd hope so.
[23:20:45] <Krenair>	 :P
[23:25:45] <Negative24>	 Krenair: I hope so as well
[23:26:16] <Negative24>	 Krenair: but I'm afraid no one is going to listen until I start pinging the wrong people and they redirect me to someone else :)
[23:27:35] <Krenair>	 Negative24, is this https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy ?
[23:28:03] <Negative24>	 maybe
[23:28:41] <Negative24>	 in that case I should ping YuviPanda and Coren
[23:31:24] <YuviPanda>	 !ask etc :)
[23:31:24] <wm-bot>	 Hi, how can we help you? Just ask your question.
[23:31:32] <YuviPanda>	 I'm outside on a bus and stuff.
[23:31:48] <Negative24>	 Well Yuvi's just traveling the world!
[23:32:03] <YuviPanda>	 Usually if a question is already out there and I know the answer I pop up and answer...
[23:32:10] <YuviPanda>	 :)
[23:32:14] <Negative24>	 Does nginx cache for project files and how do I disable it
[23:32:29] <Negative24>	 and/or purge it
[23:32:33] <Krenair>	 https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/labsproxy.pp makes it look like that is the right project
[23:32:50] <YuviPanda>	 Nope. No caching at all.
[23:33:18] <Negative24>	 That's not good cause that doesn't explain my problem
[23:33:31] <YuviPanda>	 What is the problem 
[23:34:10] <Krenair>	 L235, still there?
[23:34:38] <Negative24>	 YuviPanda: Web server on phab-02 is serving bad cached copies of css
[23:37:45] <YuviPanda>	 Negative24: have you tried hitting it from localhost to Dee if you are getting cached copies?
[23:38:54] <Negative24>	 Well its definitely not a local cache since its been reproduced by firefox, mmodell/twentyafterfour and epriestley
[23:39:37] <Negative24>	 and yes I have tried localhost