[00:03:06] <wikibugs>	 6Labs: Enable nfs to access /data/project for the wikimetrics project on labs - https://phabricator.wikimedia.org/T122258#1899730 (10madhuvishy) 3NEW a:3yuvipanda
[03:12:50] <Volker_E>	 I hope somebody could help me to connect to a labs instance
[03:13:43] <Volker_E>	 SPage was helping me last time
[03:13:58] <Volker_E>	 and I'm running into some trouble again
[03:14:20] <Volker_E>	 it's the instance for livingstyleguide.wmflabs.org/wiki/Main_Page
[03:20:52] <Volker_E>	 https://www.irccloud.com/pastebin/YezJo6f3/ssh-error
[06:47:55] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Arlitt was created, changed by Arlitt link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Arlitt edit summary: Created page with "{{Tools Access Request |Justification=I am working on the oa project and would like to access the recitation-bot project here. |Completed=false |User Name=Arlitt }}"
[09:08:15] <wikibugs>	 6Labs: New instance creation broken - https://phabricator.wikimedia.org/T122281#1900424 (10yuvipanda) 3NEW
[09:36:51] <wikibugs>	 6Labs: nfs-exports.service  is failing on labstore1001 often - https://phabricator.wikimedia.org/T122250#1900442 (10yuvipanda)
[09:39:33] <wikibugs>	 6Labs: nfs-exports.service  is failing on labstore1001 often - https://phabricator.wikimedia.org/T122250#1900443 (10yuvipanda) p:5High>3Normal I am pretty sure this is all LDAP flakiness, since this hits LDAP pretty heavily. It's also low-criticalness service, if it fails new instances in projects that have...
[09:48:09] <wikibugs>	 10PAWS, 5Patch-For-Review: PAWS network error: - https://phabricator.wikimedia.org/T120561#1900465 (10yuvipanda) 5Open>3Resolved Pretty sure the flannel fix fixed it :) Instance lockup is being tracked in T121998
[09:48:37] <wikibugs>	 6Labs: New instance creation broken - https://phabricator.wikimedia.org/T122281#1900469 (10yuvipanda) Trusty seems fine, Jessie's broken.
[09:49:55] <wikibugs>	 10PAWS: Add a way to expose static files to the internet from PAWS - https://phabricator.wikimedia.org/T119859#1900471 (10yuvipanda) I've split up the proxy and the hub into separate pods so they can be scaled separately now. Now to add register-with-proxy-support to nbserve...
[09:50:43] <wikibugs>	 6Labs: New instance creation broken - https://phabricator.wikimedia.org/T122281#1900472 (10yuvipanda) oooh, fascinating. Another instance created worked out fine. wut.
[09:51:30] <wikibugs>	 6Labs: New instance creation broken - https://phabricator.wikimedia.org/T122281#1900473 (10yuvipanda) sacrificial-puppy was trusty instance, sacrificial-kitten was jessie that didn't work and sacrificial-red-hair was jessie that does work.
[10:06:58] <shinken-wm>	 PROBLEM - Puppet failure on sacrificial-red-hair is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0]
[10:09:11] <shinken-wm>	 PROBLEM - Puppet failure on congratulatory-green-hair is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0]
[10:12:05] <shinken-wm>	 RECOVERY - Puppet failure on sacrificial-red-hair is OK: OK: Less than 1.00% above the threshold [0.0]
[10:14:03] <shinken-wm>	 RECOVERY - Puppet failure on congratulatory-green-hair is OK: OK: Less than 1.00% above the threshold [0.0]
[10:17:14] <wikibugs>	 6Labs: Enable nfs to access /data/project for the wikimetrics project on labs - https://phabricator.wikimedia.org/T122258#1900491 (10yuvipanda) 5Open>3declined So I talked to @madhuvishy in person today, and I think I've convinced her that NFS is evil and requires blood sacrifices that we do not have the cap...
[10:35:23] <wikibugs>	 6Labs: Instances locking up randomly - https://phabricator.wikimedia.org/T121998#1900510 (10faidon) So, tools-worker-07 is currently "stuck". Kernel and userland both seem to be responsive, but logins over SSH timeout. I logged in over VNC and Yuvi ran "passwd -d root" over salt (which still works), so I could l...
[10:43:09] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: strace sshd to figure out which filesystems are hit on login - https://phabricator.wikimedia.org/T104327#1900514 (10yuvipanda) Boom some more!
[10:44:58] <wikibugs>	 6Labs: Instances locking up randomly - https://phabricator.wikimedia.org/T121998#1900516 (10yuvipanda) Ok, I'm going to move tools-worker-01 to -05 to 3.19 and -06 to -09 on 4.2
[10:46:26] <YuviPanda>	 !log tools migrate tools-worker-01 to 3.19 kernel
[10:46:30] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[11:35:10] <wikibugs>	 6Labs: Instances locking up randomly - https://phabricator.wikimedia.org/T121998#1900612 (10faidon) @valhallasw raised the good point that root SSH logins should be working even with NFS failed/stuck. I ran strace and found out that ssh-key-ldap-lookup was trying to access /home/ssh-key-ldap-lookup/.local/lib/py...
[13:50:29] <The_Photographer>	 hi
[13:50:30] <The_Photographer>	 I have a project in github, how I could update the source in wmflabs automatically?
[13:57:04] <The_Photographer>	 hoo: I could change the autologout time in toolserver?
[13:57:11] <valhallasw`cloud>	 The_Photographer: I suppose you could have some sort of post-receive hook on github, but most people just deploy in a seperate step
[13:57:52] <The_Photographer>	 valhallasw`cloud: I don't know if i could do login in wmflabs from a post-receive
[13:57:54] <valhallasw`cloud>	 I don't think we have an autologout?
[13:58:11] <The_Photographer>	 ssh logout time
[13:58:30] <The_Photographer>	 autologout because the session is inactive
[13:58:32] <The_Photographer>	 ssh session
[13:58:38] <valhallasw`cloud>	 ah. Send keepalive packets.
[13:58:54] <valhallasw`cloud>	 http://www.howtogeek.com/howto/linux/keep-your-linux-ssh-session-from-disconnecting/
[13:59:43] <The_Photographer>	 thanks
[13:59:43] <valhallasw`cloud>	 The_Photographer: I don't think there's a simple way to do post-receive from github, no.
[14:00:39] <The_Photographer>	 valhallasw`cloud: do you know some tool for Continuous integration ?
[14:00:47] <The_Photographer>	 in wmflabs, of course
[14:01:08] <valhallasw`cloud>	 what are you trying to do?
[14:01:20] <valhallasw`cloud>	 (I don't see how CI is related to github post-receive hooks)
[14:02:18] <The_Photographer>	 I want my code ready to test in wmflabs for each commit in github
[14:02:46] <The_Photographer>	 I dont have time to do a git pull for test everything
[14:02:53] <The_Photographer>	 I can't test it locally
[14:03:41] <valhallasw`cloud>	 again, no easy way to do that.
[14:04:24] <valhallasw`cloud>	 you can push to a repo on tool labs, and use a post-commit hook there, but it's always fiddly.
[14:04:39] <valhallasw`cloud>	 the easiest way is to just have a terminal open and do a git pull
[14:06:44] <The_Photographer>	 ok, thanks
[14:07:12] <valhallasw`cloud>	 (or make a shell script that does that immediately after pushing; e.g.  git push && ssh tools-login.wmflabs.org "become wikibugs /bin/sh -c 'cd src/wikibugs2 && git pull'")
[14:08:22] <The_Photographer>	 yes a local script
[14:31:52] <The_Photographer>	 valhallasw`cloud: could you add gerrit for wikiradio ?
[14:32:02] <valhallasw`cloud>	 The_Photographer: no, please follow the on-wiki process.
[14:32:10] <The_Photographer>	 ok
[14:32:55] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:37:19] <valhallasw`cloud>	 hm. works for me
[14:37:43] <valhallasw`cloud>	 it's mostly just flaky, it seems
[14:37:52] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 966126 bytes in 8.114 second response time
[14:42:27] <valhallasw`cloud>	 !log tools toollabs homepage is unhappy because tools.xtools-articleinfo is using a lot of cpu on tools-webgrid-lighttpd-1409. Checking to see what's happening there.
[14:42:31] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[14:43:08] <andrewbogott>	 valhallasw`cloud: thank you for investigating!  I’m still hoping to eat breakfast before I get in to this :)
[14:43:46] <valhallasw`cloud>	 it looks like someone is scraping GET /xtools-articleinfo/?article=Category:University_of_California,_San_Diego&project=en.wikipedia.org&uselang=*
[14:47:57] <valhallasw`cloud>	 and that IP is also firing lots of requests to geohack
[14:48:53] <Coren>	 valhallasw`cloud: Wouldn't be the first idiot spider that crawls internal links into a loop.
[14:49:18] <valhallasw`cloud>	 if it's a spider, it's also malicious -- the user agent is "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit (KHTML, like Gecko)"
[14:49:24] <valhallasw`cloud>	 (also not too uncommon)
[14:50:32] <Coren>	 Also.
[14:50:51] <Coren>	 What does the IP whois to?
[14:51:04] <valhallasw`cloud>	 ipXX-YY-ZZ-AA.sd.sd.cox.net
[14:51:32] <valhallasw`cloud>	 sorry, whois. Eh, cox communications ;-)
[14:52:09] <Coren>	 Enduser.  Huh.  Either a zombie, or some client "read offline" insanity.  I'm pretty sure safari doesn't do that by default.
[14:52:23] <Coren>	 I'm betting the former
[14:52:27] <valhallasw`cloud>	 google suggests it's apple mail, which makes even less sense
[14:52:56] <Coren>	 I'm thinking someone with a email scraper bit of malware.
[14:53:14] <Coren>	 I'd blackhole at the proxy, at least for a while.
[14:57:35] <The_Photographer>	 How I could add a remote github repository?
[14:58:03] <The_Photographer>	 git remote add origin git@github.com:emijrp/wikiradio.git -->> Please make sure you have the correct access rights
[14:58:11] <The_Photographer>	 I am in my public_html tool folder
[15:07:20] <valhallasw`cloud>	 Coren: any issues with having the IP address in puppet?
[15:07:36] <valhallasw`cloud>	 and if so, how can we set them out-of-bound on both servers?
[15:23:39] <valhallasw`cloud>	 andrewbogott: ^?
[15:34:49] * Coren ponders.
[15:35:19] <Coren>	 valhallasw`cloud: It can't be associated with a user, so it's not PII.
[15:35:42] <Coren>	 valhallasw`cloud: There's a bit of BEANS, but I don't think that can't be helped.  At least, not without uglies.
[15:36:01] * Coren counts negatives.
[15:36:03] <valhallasw`cloud>	 BEANS?
[15:36:32] <Coren>	 valhallasw`cloud: https://en.wikipedia.org/wiki/Wikipedia:BEANS
[15:36:39] <Coren>	 Sorry, enwiki alphabet soup.  :-)
[15:37:14] <valhallasw`cloud>	 I'm still fiddling with nginx to get it to actually return sane data so chrome doesn't puke. Fun.
[15:38:13] <valhallasw`cloud>	 ah, extra newlines. I can do that.
[16:24:48] <The_Photographer>	 hi
[16:26:00] <Ebe123>	 Hello!
[16:49:54] <andrewbogott>	 valhallasw`cloud: I’m back, sorry — are you still fighting with tools.xtools-articleinfo ?
[16:50:07] <valhallasw`cloud>	 andrewbogott: fighting with getting nginx to do what I want, mostly
[16:50:26] <valhallasw`cloud>	 'I'll make sure they get a nice 403, that can't be too hard?'
[16:50:33] <valhallasw`cloud>	 #famouslastwords
[16:50:53] <andrewbogott>	 ok — let me know if I can help :)
[16:53:55] <The_Photographer>	 Hellow
[16:54:34] <andrewbogott>	 The_Photographer: what’s up?
[16:54:39] <The_Photographer>	 how I could solve this https://tools.wmflabs.org/wikiradio/get_content.html problem with CORS, I am trying make a test 
[16:54:57] <The_Photographer>	 thanks andrewbogott 
[16:57:07] <andrewbogott>	 The_Photographer: Ah, dang, I’m probably not the right person to ask.  Shouting out the question here is a reasonable start… you could also write to labs-l
[16:57:31] <The_Photographer>	 I am sorry
[16:58:15] <andrewbogott>	 You’re not in the wrong place :)  A few of us here can troubleshoot labs infrastructure (that’s me) but there are also often tools developer folks as well
[16:59:42] <valhallasw`cloud>	 andrewbogott: https://tools.wmflabs.org/.error/banned.html :-D
[16:59:51] <valhallasw`cloud>	 ok, now let me puppetize this
[17:00:58] <andrewbogott>	 That’s a very polite page!  I assume that the linked page is a wip and not that you meant a different link?
[17:01:26] <valhallasw`cloud>	 andrewbogott: writing content on that page is next on the to-do
[17:01:30] <The_Photographer>	 I am sorry, why you are banned  :(
[17:01:52] <valhallasw`cloud>	 The_Photographer: it's an example page, not normally shown
[17:02:06] <The_Photographer>	 ok
[17:14:42] <valhallasw`cloud>	 andrewbogott: what's your assessment on having the banned ip addresses in puppet/phabricator?
[17:15:13] <valhallasw`cloud>	 I *think* it's possible to use a seperate unpuppetized file, but it's a bit of a mess
[17:15:38] <andrewbogott>	 valhallasw`cloud: it doesn’t freak me out that much but… is this really the first time we’ve ever banned an IP?
[17:15:41] <andrewbogott>	 Coren: ^ ?
[17:16:01] <Coren>	 andrewbogott: No.  It's the first time we did it cleanly in puppet though.
[17:16:19] <Coren>	 andrewbogott: Every other time was emergency filtering directly in iptables because outage-causing.
[17:16:32] <andrewbogott>	 ah, ok
[17:16:36] <Coren>	 (Also, before clean ferm rules)
[17:16:49] <andrewbogott>	 well… I don’t think that putting an IP in puppet leaks any information other than ‘this is an IP address'
[17:17:08] <wikibugs>	 6Labs, 10Tool-Labs: 70.179.6.X is hammering xtools-articleinfo - https://phabricator.wikimedia.org/T122307#1901159 (10valhallasw) 3NEW
[17:17:13] <valhallasw`cloud>	 ok, then I'll just hop it in there.
[17:17:25] <Coren>	 andrewbogott: That was my opinion as well - it's not PII since it is not linked to any user.
[17:18:00] <andrewbogott>	 Probably good to send an email to labs-l saying that you blocked it, so that potential innocent victims know what is happening
[17:18:11] <valhallasw`cloud>	 *nod*
[17:18:18] <valhallasw`cloud>	 although that error page should be clear enough ;-)
[17:19:40] <andrewbogott>	 oh, that’s true
[17:22:36] <The_Photographer>	 guys i am using the example from https://www.mediawiki.org/wiki/Manual:CORS in https://tools.wmflabs.org/wikiradio/get_content.html whats happend?
[17:23:29] <The_Photographer>	 it show "Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://en.wikipedia.org/w/api.php?action=query&meta=tokens&format=json&origin=https%3A%2F%2Fwww.mediawiki.org. (Reason: CORS header 'Access-Control-Allow-Origin' missing)."
[17:25:02] <The_Photographer>	 I added to .htaccess t"Header set Access-Control-Allow-Origin "*""
[17:25:04] <andrewbogott>	 Coren: does my question on https://gerrit.wikimedia.org/r/#/c/257411/ make sense or am I misunderstanding what that patch does?
[17:25:20] * Coren goes read said question.
[17:27:25] <Coren>	 The question makes sense, but the answer is "Nah."  :-)  Just explained on the patch.
[17:29:04] <Coren>	 (But the patch is outdated for another reason)
[17:29:51] <The_Photographer>	 how I could red a wikipedia article content from php? where I could begin?
[17:30:11] <Coren>	 The_Photographer: You want to use the API for that.
[17:30:23] <The_Photographer>	 Coren: show me the way
[17:30:40] <The_Photographer>	 Coren: from a wmflabs applicatio
[17:30:41] <The_Photographer>	 n
[17:31:12] <Coren>	 The_Photographer: That's rather a wide scope.  :-)  Lemme see if I can find some example PHP code for you.
[17:31:20] <The_Photographer>	 thanks
[17:35:05] <The_Photographer>	 I am trying do it with javascript, however its showing a  CORS problem, then, php could be more easy
[17:41:08] <DispenserAFK>	 Could somebody DROP DATABASE u2815_p;?  I'm not using it--and can't even touch it anymore due security permission changes.
[17:41:48] * Dispenser taps Coren on the shoulder
[17:41:55] <valhallasw`cloud>	 Dispenser: please create a task in phab
[17:42:01] <valhallasw`cloud>	 #DBA project
[17:51:20] <andrewbogott>	 valhallasw`cloud: are you ready for me to merge that now?
[17:51:24] <valhallasw`cloud>	 andrewbogott: yep!
[17:51:58] <andrewbogott>	 ‘abr: always be rebasing'
[17:52:02] <valhallasw`cloud>	 :D
[17:52:14] <valhallasw`cloud>	 'git abr' sounds like something that could exist
[17:52:28] <valhallasw`cloud>	 watch 'git fetch && git rebase origin/production'
[17:54:04] <andrewbogott>	 Yeah, we could turn git-review into a service that runs in the background, detects a change in the tip and automatically resubmits patches.  The spam potential would be limitless!
[17:55:27] <andrewbogott>	 merged
[17:57:37] <mutante>	 service paladox start
[17:58:33] * valhallasw`cloud prods labs-puppetmaster
[18:01:41] <jynus>	 Dispenser, please create a phabricator task to verify you are the owner of that database. I can rename it for you, but I have to confirm it is owned by you
[18:02:18] <The_Photographer>	 how I could get a wikipedia article content?
[18:03:02] <The_Photographer>	 In any language
[18:03:07] <The_Photographer>	 I don't care
[18:03:19] <The_Photographer>	 I need only need the article content
[18:03:27] <andrewbogott>	 The_Photographer: I’ve had good luck using https://github.com/mwclient/mwclient
[18:03:28] <andrewbogott>	 in python
[18:03:35] <mutante>	 The_Photographer: https://en.wikipedia.org/w/api.php?action=parse&page=foobar
[18:03:45] <andrewbogott>	 There’s also ‘pywikibot’ which I think does similar things, but I’m unsure if that’s the right scope
[18:04:40] <mutante>	 it also depends if you want wiki code or HTML 
[18:05:31] <Dispenser>	 http://dispenser.homenet.org/~dispenser/sources/make_dabs_gz.sh Script to mass download parsed disambiguation pages and package em into a tar file
[18:06:17] <mutante>	 The_Photographer: here's variety for plain text https://en.wikipedia.org/w/api.php?format=xml&action=query&prop=extracts&titles=Cat&redirects=true
[18:06:22] <The_Photographer>	 mutante: the raw code
[18:06:25] <mutante>	 mostly
[18:06:37] <The_Photographer>	 mutante: I can't use the api
[18:06:55] <The_Photographer>	 because CORS problem
[18:07:08] <Dispenser>	 There's also ?action=render
[18:07:12] <mutante>	 wouldnt that be the same problem whether you use the API or not?
[18:07:13] <The_Photographer>	 andrewbogott: some idea to install it in wmflabs?
[18:07:43] <The_Photographer>	 mutante: I can't use the api from javascript
[18:07:43] <andrewbogott>	 valhallasw`cloud: ok, that’s merged finally
[18:07:51] <mutante>	 The_Photographer: so you want the mediawiki syntax?
[18:08:04] <mutante>	 as you would use it when editing a page?
[18:08:16] <The_Photographer>	 mutante: I don't care, I use this page like javascript
[18:08:25] <Dispenser>	 The_Photographer: Yes you can.  http://dispenser.homenet.org/~dispenser/view/Dab_solver does it all the time
[18:08:26] <mutante>	 The_Photographer: http://stackoverflow.com/questions/1625162/get-text-content-from-mediawiki-page-via-api
[18:08:42] <Dispenser>	 You need to setup callbacks
[18:09:18] <Dispenser>	 https://en.wikipedia.org/w/api.php?action=parse&format=jsonfm&callback=mosdab_callback&redirects&page=/\
[18:09:50] <The_Photographer>	 mutante: I am sorry, I cant use the fucking api because CORS problem
[18:10:11] <mutante>	 The_Photographer: i think Dispenser has the solution to that 
[18:10:24] <The_Photographer>	 https://tools.wmflabs.org/wikiradio/get_content.html
[18:11:16] <The_Photographer>	 Dispenser: I am using the same example from https://www.mediawiki.org/wiki/Manual:CORS and it simply dont work
[18:11:30] <Dispenser>	 http://dispenser.homenet.org/~dispenser/view/Dabfix#Statistics That table's loaded remotely
[18:11:59] <Dispenser>	 <script type="text/javascript">function cacheAPIcall(rspn){document.getElementById("replacementme").innerHTML=rspn.parse.text['*'];}</script>
[18:12:01] <Dispenser>	 <script type="text/javascript" src="https://en.wikipedia.org/w/api.php?action=parse&amp;format=json&amp;callback=cacheAPIcall&amp;redirects&amp;page=User:Dispenser/MOS:DAB_stats&amp;smaxage=2592000"></script>
[18:13:29] <The_Photographer>	 Dispenser: please, could you use pastebin?
[18:14:06] <The_Photographer>	 because I am not underestanding you
[18:14:22] <mutante>	 it's just about the URL in there
[18:14:30] <mutante>	 the api.php link uses callback=
[18:14:33] <mutante>	 that's the difference
[18:14:50] <The_Photographer>	 I am using callback
[18:15:08] <mutante>	 sorry, then i don't know
[18:15:20] <The_Photographer>	 no problem
[18:16:20] <The_Photographer>	 Anyway, I preffer do it from server side
[18:16:23] <Dispenser>	 The_Photographer: http://pastebin.com/eLnf2AGq
[18:17:22] <andrewbogott>	 valhallasw`cloud: working now?
[18:17:36] <valhallasw`cloud>	 andrewbogott: almost. For loops doing something else than I expected
[18:17:49] <valhallasw`cloud>	 andrewbogott: https://gerrit.wikimedia.org/r/260775 fixed that -- I think :/
[18:18:00] <valhallasw`cloud>	 but puppet runs correctly, and the .error subdir works
[18:18:12] <valhallasw`cloud>	 just the list of IPs that fails to display
[18:18:14] <valhallasw`cloud>	 hm
[18:18:30] <valhallasw`cloud>	 there's also no 1, though, so that suggests the for loop fails to run at all
[18:18:33] <andrewbogott>	 oops, yeah, that looks necessary
[18:18:45] <The_Photographer>	 Dispenser: thanks
[18:20:19] <valhallasw`cloud>	 andrewbogott: I'm going to make dinner and look at it later
[18:20:27] <The_Photographer>	 see ya
[18:20:46] <andrewbogott>	 valhallasw`cloud: ok, I’ll be around on and off for a while
[18:20:47] <Dispenser>	 So was it what you were looking for?
[18:21:31] <The_Photographer>	 Dispenser: do you have some documentation?
[18:21:47] <The_Photographer>	 Dispenser: well, I am looking for ajax
[18:21:56] <Dispenser>	 I just read through https://en.wikipedia.org/w/api.php?action=help&recursivesubmodules=1
[18:22:18] <Dispenser>	 What are you specifically looking for?
[18:22:57] <Dispenser>	 BTW, AJAX doesn't work cross domain.  You need to create a new <script> element
[18:23:06] <The_Photographer>	 Dispenser: thats the problem
[18:23:07] <Dispenser>	 JSONP is what you're looking for
[18:24:42] <The_Photographer>	 Dispenser: could you take a look to my source -->> https://tools.wmflabs.org/wikiradio/get_content.html
[18:24:46] <The_Photographer>	 its very simple
[18:25:39] <Dispenser>	 https://learn.jquery.com/ajax/working-with-jsonp/ dataType: "jsonp",
[18:26:01] <Dispenser>	 I need to learn JQuery
[18:28:26] <The_Photographer>	 ok
[18:31:26] <Dispenser>	 The_Photographer: All JSONP response use an anonymous userid, so the edittoken will always be +\.  You need OAuth or the weird little contraption I made years before WMF "blessed" us with OAuth
[18:32:26] <The_Photographer>	 great
[18:34:43] <Dispenser>	 So this is suppose to just install a user script?   Good luck getting users passed the scary OAuth dialogue.
[18:36:14] <valhallasw`cloud>	 andrewbogott: dynamicproxy:banned_ips vs dynamicproxy::banned_ips! And it's working now :-)
[18:37:25] <Dispenser>	 Or even getting OAuth running.  I've been trying for the last 6 months to reduce permissions on my OAuth application with each expiring
[18:38:46] <valhallasw`cloud>	 !log tools dynamicproxy ban system deployed on tools-proxy-02 working correctly for localhost; switching over users there by moving the external IP.
[18:38:50] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[18:40:18] <valhallasw`cloud>	 !log tools scratch that, first going to eat dinner
[18:40:23] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[18:59:39] <valhallasw`cloud>	 !log tools switched to -02, worked correctly, switched back. Switching back does not seem to fully work?!
[18:59:43] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[19:01:11] <valhallasw`cloud>	 !log tools ah, connections that are kept open. A new incognito window is routed correctly.
[19:01:15] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[19:04:02] <The_Photographer>	 I hate OAutg
[19:04:04] <The_Photographer>	 h
[19:08:57] <valhallasw`cloud>	 Ugh, you suppress one scraper and you get another one in return. 
[19:11:03] <andrewbogott>	 valhallasw`cloud: great, so we’re done?
[19:11:15] <andrewbogott>	 oh, crap, I missed the very last thing you said :)
[19:11:51] <valhallasw`cloud>	 andrewbogott: apart from the 'actually figuring out which clients to ban': yes :-)
[19:12:01] <valhallasw`cloud>	 this one is chrome, probably some sort of preloader plugin
[19:12:06] <valhallasw`cloud>	 or juts chrome being chrome? not sure
[19:15:07] <valhallasw`cloud>	 well, my chrome doesn't
[19:15:08] <valhallasw`cloud>	 I
[19:15:29] <valhallasw`cloud>	 I'm wondering whether the solution should maybe be found in xtools rather than in getting people with broken clients to be less... broken
[19:16:23] <Coren>	 Some rate limiting for infinitely traversable pages would be a Good Thing, of course.
[19:16:58] <Volker_E>	 another request for help, http://mwui.wmflabs.org/ returns a 500 
[19:17:02] <Coren>	 But that requires the author to fiddle with their code; it's not something we can slap on a posteriori.
[19:17:03] <valhallasw`cloud>	 right. The page takes 4 seconds to generate, and it's being scraped at that same rate
[19:17:40] <valhallasw`cloud>	 (pages that take 4 seconds to generate should be behind a POST, if you ask me)
[19:17:59] <The_Photographer>	 Dispenser: do you have some idea to do it from server side?
[19:18:14] <Dispenser>	 OAuth
[19:20:49] <The_Photographer>	 I could create a robot account
[19:20:55] <The_Photographer>	 ?
[19:24:36] <Dispenser>	 If I understand what you're trying to do.  User JS and CSS are protected files only editable by admins.
[19:26:36] <Dispenser>	 Actually what are you trying to do?  The_Photographer
[19:26:51] <The_Photographer>	 I am trying load a meta page content to assing it to a javascript var
[19:27:24] <The_Photographer>	 I could load the page in server side 
[19:27:31] <The_Photographer>	 load meta page content in server side
[19:27:37] <Dispenser>	 So a JSON parser
[19:28:33] <The_Photographer>	 yes
[19:29:11] <The_Photographer>	 do you know some server side script to load a page content?
[19:29:15] <wikibugs>	 6Labs, 10Tool-Labs: Block xtools-articleinfo from scrapers - https://phabricator.wikimedia.org/T122327#1901595 (10valhallasw) 3NEW
[19:30:01] <The_Photographer>	 I don't like the dirty thought of cacheAPIcall
[19:35:59] <Dispenser>	 https://meta.wikimedia.org/w/api.php?action=query&prop=revisions&titles=Wikiradio_%28tool%29/playlist/es&rvprop=content&callbackFunc&format=json&formatversion=2
[19:36:01] <Dispenser>	 function callbackFunc(obj) { DoSomething( JSON.parse( obj.query.pages.revisions[0].content ) ); }
[19:36:18] <Dispenser>	 The_Photographer: I think that should get you started
[19:36:38] <Dispenser>	 s/DoSomething/window.alert/
[19:37:32] <The_Photographer>	 Dispenser: yes, I underestand, however, I was thinking in do a server side (with php for example) to get the page content
[19:38:00] <The_Photographer>	 to do some ajax request 
[19:38:12] <Dispenser>	 Same thing
[19:38:56] <The_Photographer>	 I could use file_get_contents with this url?
[19:40:51] <The_Photographer>	 I don't know if I'm doing something illegal
[19:41:36] <Dispenser>	 # Quick + Dirty python code
[19:41:38] <Dispenser>	 import requests; print requests.get('https://meta.wikimedia.org/w/api.php?action=query&prop=revisions&titles=Wikiradio_%28tool%29/playlist/es&rvprop=content&callbackFunc&format=json&formatversion=2').json()['query']['pages'][0]['revisions'][0]['content']
[19:43:05] <The_Photographer>	 interesting
[19:47:12] <valhallasw`cloud>	 Dispenser: please set your user-agent ;-)
[19:54:41] <The_Photographer>	 valhallasw`cloud: yes xD
[19:55:44] <The_Photographer>	 maybe Mozilla
[19:56:08] <valhallasw`cloud>	 .... no, something that clarifies it's your tool making the request
[19:56:39] <The_Photographer>	 I was using "User-Agent: Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.102011-10-16 20:23:10\r\n" // i.e. An iPad
[19:56:47] <The_Photographer>	 What it could be?
[19:56:50] <Earwig>	 why would you fake it? you're not an iPad
[19:57:09] <The_Photographer>	 Earwig: because its a sample of http://stackoverflow.com/questions/2107759/php-file-get-contents-and-headers
[19:57:12] <valhallasw`cloud>	 The_Photographer: 'User-Agent: <MyToolName> by The_Photographer [[es:User:...]]'
[19:57:23] <The_Photographer>	 valhallasw`cloud: thanks
[19:58:25] <Dispenser>	 We probably should spend some time correcting misinformation on stack overflow...
[20:07:36] <The_Photographer>	 Dispenser: https://tools.wmflabs.org/wikiradio/playlist.php
[20:07:38] <The_Photographer>	 :)
[20:08:19] <The_Photographer>	 now I can use simple ajax
[20:09:41] <Dispenser>	 You do know the JS code I gave you had a few errors (I didn't transverse the tree correctly) but did the same thing
[20:17:17] <The_Photographer>	 well, yes, however, server side proxy could help with anothers thing to do a better code
[20:32:17] <The_Photographer>	 Dispenser: its a simple proxy, example: https://tools.wmflabs.org/wikiradio/playlist.php?name=es is loading really this page https://meta.wikimedia.org/wiki/Wikiradio_(tool)/playlist/es
[20:32:33] <The_Photographer>	 I don't know if I need write some comment about the importance of this page in meta?
[20:33:29] <The_Photographer>	 valhallasw`cloud: And yes, I added the agent 
[20:33:43] <Dispenser>	 It add  unnecessary complexity.  This generally make it harder to debug or cause more error conditions.
[20:34:25] <The_Photographer>	 Dispenser: I underestand your point
[20:34:38] <Dispenser>	 Is probably a good idea considering how delete button happy admins on enwp are
[20:35:17] <Dispenser>	 Write a page somewhere else linking to it so they can find it in the back links list
[20:35:44] * Dispenser mutters something about the GeoHack template page being deleted
[20:37:08] <wikibugs>	 6Labs, 10Tool-Labs, 10DBA: Rename user database - https://phabricator.wikimedia.org/T122338#1901847 (10Dispenser) 3NEW
[20:41:46] <The_Photographer>	 Dispenser: yes, I am doing it
[20:42:01] <The_Photographer>	 Dispenser: I don't know if meta is the right palce
[20:42:02] <The_Photographer>	 place
[20:42:53] <Dispenser>	 If it single language only tool, then put on your home wiki so you can see it in your watchlist
[21:04:46] <YuviPanda>	 omg, so much scrollback
[21:04:57] <YuviPanda>	 valhallasw`cloud: andrewbogott is the accidental DoS done?
[21:05:05] <YuviPanda>	 we should probably use iptables / ferm for these than nginx
[21:05:08] <valhallasw`cloud>	 YuviPanda: yeah
[21:05:21] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1901987 (10mobrovac) 3NEW
[21:05:24] <valhallasw`cloud>	 and no, not really? the issue is it's killing the lighttpd backend
[21:05:25] <valhallasw`cloud>	 nginx is fine
[21:05:57] <YuviPanda>	 valhallasw`cloud: hmm, I guess error page is ok
[21:06:02] <YuviPanda>	 valhallasw`cloud: <3 thanks for doing that
[21:06:04] <valhallasw`cloud>	 YuviPanda: it's also pretty!
[21:06:11] <valhallasw`cloud>	 YuviPanda: https://tools.wmflabs.org/.error/banned.html
[21:06:12] <YuviPanda>	 andrewbogott: did you see the updates on the hanging instances issue?
[21:06:39] <valhallasw`cloud>	 served with a 403 and /me's best wishes
[21:06:53] <YuviPanda>	 :D
[21:06:58] <YuviPanda>	 yeah agreed that's better
[21:07:33] <YuviPanda>	 now to change kernels
[21:07:39] <YuviPanda>	 I suppose I should just create new machines instead
[21:08:47] <valhallasw`cloud>	 do you need proxy-02? I'm fiddling to see if we can do letsencrypt for other domains
[21:08:58] <valhallasw`cloud>	 basically trying to see if I can get nginx to SNI
[21:09:01] <YuviPanda>	 valhallasw`cloud: why SNI?
[21:09:18] <YuviPanda>	 and what do you mean by other domains
[21:09:32] <YuviPanda>	 if you're looking at *.tools.wmflabs.org, I think we can afford to buy that wildcard specifically if need be
[21:09:43] <valhallasw`cloud>	 YuviPanda: no, I mean e.g. that lizenzgenerator
[21:10:09] <YuviPanda>	 oh. but that's not on tools, right?
[21:10:28] <valhallasw`cloud>	 the initial version was, and I think they just need a web server, so tools would be a sane place
[21:10:48] <valhallasw`cloud>	 I'm also thinking of e.g. pywikibot.org / pywikipedia.org
[21:10:57] <valhallasw`cloud>	 which are now being redirected, but no https
[21:11:02] <YuviPanda>	 ah I see
[21:11:14] <valhallasw`cloud>	 well, the first isn't, it's still in puppet waiting for an OK from ops
[21:11:18] <YuviPanda>	 hmm, so labs proxy is setup similarly
[21:11:25] <YuviPanda>	 so same code should work here I guess?
[21:11:26] <valhallasw`cloud>	 also I want to learn letsencrypt
[21:11:33] <YuviPanda>	 heh
[21:11:35] <YuviPanda>	 e
[21:11:50] <YuviPanda>	 yeah +1 but I would prefer if you not use proxy-02....
[21:11:51] <valhallasw`cloud>	 labs proxy doesn't do different virtualhosts with different certs, though
[21:12:00] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1902019 (10Dzahn) a:3Dzahn
[21:12:00] <shinken-wm>	 PROBLEM - Host sacrificial-puppy is DOWN: CRITICAL - Host Unreachable (10.68.20.104)
[21:12:04] <YuviPanda>	 or if you do, you should delete it and clean it up afterwards
[21:12:07] <YuviPanda>	 err
[21:12:10] <YuviPanda>	 recreate it
[21:12:55] <YuviPanda>	 sacirifical puppy was me
[21:13:22] <valhallasw`cloud>	 I've been using proxy-02 all day to figure out that 403 mess :-p
[21:13:34] <valhallasw`cloud>	 so maybe I /should/ just spin up a third one
[21:13:35] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1901987 (10Dzahn) I checked /mnt/jenkins-workspace/puppet-compiler/output# for especially large ones, and:  root@compiler02:/mnt/jenkins...
[21:13:51] <valhallasw`cloud>	 YuviPanda: wanna do the new kernel on that one, then?
[21:14:50] <YuviPanda>	 valhallasw`cloud: hmm I just got a 'failed to create instance' so figuring out what's going on there
[21:14:54] <valhallasw`cloud>	 :{
[21:15:02] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1902035 (10Dzahn) 5Open>3Resolved these really large ones must have been running on a LOT of nodes (maybe it was attempted to use th...
[21:16:26] <shinken-wm>	 PROBLEM - Host sacrificial-red-hair is DOWN: CRITICAL - Host Unreachable (10.68.20.92)
[21:16:31] <YuviPanda>	 valhallasw`cloud: ah, just needs a quota bump
[21:17:12] <YuviPanda>	 valhallasw`cloud: actually, you can keep playing with proxy-02, I think.
[21:17:17] <valhallasw`cloud>	 ok
[21:17:19] <YuviPanda>	 valhallasw`cloud: it's probably fine
[21:17:36] <YuviPanda>	 valhallasw`cloud: also we might need it to be 4.2, since 3.19 was showing a lot of ksoftirqd traffic
[21:17:44] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6operations, 7Puppet: compiler02.puppet3-diffs.eqiad.wmflabs out of disk space - https://phabricator.wikimedia.org/T122346#1902074 (10Dzahn)
[21:17:47] <YuviPanda>	 ugh, it'll be pretty sad if 3.19 has the ksoftirqd issue and 4.2 has NFS issue
[21:17:52] <YuviPanda>	 valhallasw`cloud: but the proxies have no NFS anymore
[21:17:54] <YuviPanda>	 so
[21:18:04] <valhallasw`cloud>	 -01 is still on NFS
[21:18:12] <valhallasw`cloud>	 at least, /home is mounted
[21:18:17] <YuviPanda>	 oh? I thought I removed it...
[21:18:19] <valhallasw`cloud>	 we can just unmount all NFS-es, I guess?
[21:18:20] <YuviPanda>	 can you unmount?
[21:18:28] <valhallasw`cloud>	 ya
[21:18:50] <andrewbogott>	 YuviPanda: was the stuff about hanging instances just in the backscroll?  Or was there info someplace else?
[21:18:58] <andrewbogott>	 (I don’t think I’ve read my phab mail for the day yet)
[21:19:20] <valhallasw`cloud>	 !log tools tools-proxy-01: umount /home /data/project /data/scratch /public/dumps
[21:19:21] <YuviPanda>	 andrewbogott: it was in _security yesterday night and bits of PM, but largely it was on phab
[21:19:24] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[21:19:38] <YuviPanda>	 valhallasw`cloud: run puppet to make sure it doesn't remount?
[21:19:46] <valhallasw`cloud>	 sí.
[21:19:58] <andrewbogott>	 YuviPanda: I’ll catch up shortly
[21:20:29] <YuviPanda>	 andrewbogott: ok. tl dr is we have mostly identified the cause.
[21:20:44] <valhallasw`cloud>	 YuviPanda: looking good!
[21:21:22] <YuviPanda>	 !log tools deleted tools-worker-01 to -05, creating tools-worker-1001 to 1005
[21:21:26] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[21:22:16] <andrewbogott>	 YuviPanda: there’s the bit where faidon says "since we don't mount with "soft" for some reason"
[21:22:22] <andrewbogott>	 is that something that’s easy to fix?
[21:22:39] <YuviPanda>	 andrewbogott: presumably, although I don't know why we don't mount with soft, or what 'soft' is?
[21:22:41] <YuviPanda>	 Coren: ^
[21:22:50] <andrewbogott>	 yeah, ok, me neither :)
[21:23:06] <YuviPanda>	 yeah
[21:23:45] <shinken-wm>	 PROBLEM - Host tools-worker-04 is DOWN: CRITICAL - Host Unreachable (10.68.17.0)
[21:23:55] <valhallasw`cloud>	 YuviPanda: soft/hard is whether the a file system call fails (errors) or hangs until NFS responds
[21:24:17] <wikibugs>	 6Labs, 5Patch-For-Review: PoC bare-metal server allocation in labs -- bootstrap mode - https://phabricator.wikimedia.org/T95185#1902114 (10Andrew)
[21:24:19] <wikibugs>	 6Labs, 10netops, 6operations: Create labs baremetal subnet? - https://phabricator.wikimedia.org/T121237#1902111 (10Andrew) 5Open>3declined a:3Andrew This isn't needed.
[21:24:21] <valhallasw`cloud>	 the problem with soft is that not all software checks for return values in fs calls
[21:24:43] <shinken-wm>	 PROBLEM - Host tools-worker-05 is DOWN: CRITICAL - Host Unreachable (10.68.19.31)
[21:26:33] <shinken-wm>	 PROBLEM - Host congratulatory-green-hair is DOWN: CRITICAL - Host Unreachable (10.68.20.55)
[21:27:00] * valhallasw`cloud eyes YuviPanda
[21:27:07] <YuviPanda>	 what
[21:27:14] <YuviPanda>	 it was a test instance!
[21:27:20] <YuviPanda>	 with a clear descriptive name
[21:27:22] <shinken-wm>	 PROBLEM - Host tools-worker-03 is DOWN: CRITICAL - Host Unreachable (10.68.18.154)
[21:27:26] <valhallasw`cloud>	 :D
[21:28:56] <valhallasw`cloud>	 YuviPanda: ok, this could work! coolcoolcool.
[21:29:08] <valhallasw`cloud>	 after 15 minutes of fiddling: curl: (60) SSL certificate problem: self signed certificate
[21:29:20] <valhallasw`cloud>	 so the nginx end is easy-ish
[21:29:26] <shinken-wm>	 PROBLEM - Host tools-worker-02 is DOWN: CRITICAL - Host Unreachable (10.68.18.112)
[21:29:40] <YuviPanda>	 valhallasw`cloud: I'll be curious how we integrate letsencrypt into puppet
[21:29:50] <YuviPanda>	 valhallasw`cloud: all the betacluster people will be immensely grateful if you figure that out :P
[21:29:55] <valhallasw`cloud>	 :)
[21:30:00] <shinken-wm>	 PROBLEM - Host tools-worker-01 is DOWN: CRITICAL - Host Unreachable (10.68.18.94)
[21:30:12] <valhallasw`cloud>	 should be relatively easy if we don't hit any rate limiting
[21:30:44] <YuviPanda>	 yeah
[21:31:06] <Coren>	 "sacrificial-puppy"
[21:31:30] <Coren>	 YuviPanda: The NFS soft option is short for "I don't like my data, please corrupt it at the least issue".
[21:31:44] <YuviPanda>	 Coren: so we should use soft and run mongodb on top?
[21:32:19] <Coren>	 YuviPanda: More precisely, it allows timeout on NFS operations rather than retry until the server returns - which means the operation is lost.
[21:33:46] <Coren>	 YuviPanda: from the man page: "NB: A so-called "soft" timeout can cause silent data corruption in certain cases. As such, use
[21:33:46] <Coren>	                       the  soft option only when client responsiveness is more important than data integrity."
[21:33:59] * YuviPanda nods
[21:34:04] <YuviPanda>	 seems like a bad idea
[21:34:37] <Coren>	 Also experience allows me to expand "in certain cases" to "randomly, and for minor issues"
[21:34:46] <Coren>	 So yeah.  :-)
[21:35:07] <YuviPanda>	 Coren: did you see the ticket with the debug info about the locking up instances?
[21:35:14] <YuviPanda>	 it's probably a bug in 4.2 kernel's client nfs
[21:35:50] <Coren>	 Yeah; it still bugs me a little because if that's the case it means that we had some instances going dead for unrelated reasons as a coincidence.  I don't like coincidences.
[21:36:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-worker-1004 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0]
[21:36:38] <Coren>	 (afaik, we have no -exec- nodes with 4.2 kernels)
[21:36:58] <Coren>	 Ooo.  The dead bird is ready.  BBIAB, I need to get it out of the oven.
[21:41:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:43:05] <YuviPanda>	 valhallasw`cloud: btw, I wrote a publishing mechanism for PAWS :)
[21:43:12] <YuviPanda>	 https://github.com/yuvipanda/nbserve
[21:46:21] <The_Photographer>	 where I can add the Notes to a tool?
[21:48:49] <andrewbogott>	 YuviPanda: is the solution likely to involve building a new jessie base image?  Because if so I should probably get that underway soon, about to vanish until Tuesday.
[21:48:51] <The_Photographer>	 toolinfo.json
[21:49:03] <shinken-wm>	 PROBLEM - Puppet failure on tools-worker-1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[21:49:25] <YuviPanda>	 andrewbogott: no, I think it's fine  since the current default jessie image is on 3.16 which is stable enough
[21:50:04] <andrewbogott>	 YuviPanda: ah, it’s older jessie builds that have the bug?
[21:50:09] <andrewbogott>	 (Maybe I’m totally misunderstanding)
[21:50:23] <YuviPanda>	 andrewbogott: no, so the instances that had it were 4.2 which is a newer jessie
[21:50:25] <YuviPanda>	 newest even
[21:50:37] <YuviPanda>	 and I needed for k8s at least a 3.18 kernel, and our 3.19 kernel package had a bug in it
[21:50:39] <YuviPanda>	 so I used 4.2
[21:50:52] <YuviPanda>	 so only the k8s nodes were affected by this particular one
[21:50:52] <andrewbogott>	 ah, so you had upgraded them in place after the instances built?
[21:50:55] <YuviPanda>	 yes
[21:51:00] <andrewbogott>	 (I’m kind of surprised that that worked at all!)
[21:51:08] <andrewbogott>	 ok
[21:51:24] <andrewbogott>	 so that explains the k8s issues and… then what about the exec node freeze-ups?
[21:53:26] <YuviPanda>	 andrewbogott: right, so no idea about those :)
[21:53:33] <YuviPanda>	 andrewbogott: what, upgrading kernels? that's always worked
[21:53:53] <YuviPanda>	 andrewbogott: and works for 4.2 too, just has a random NFS client but
[21:53:55] <YuviPanda>	 *bug
[21:53:58] <andrewbogott>	 I just remember in the early days users trying to move their precise instances to oneiric and them never working again
[21:54:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:54:06] <andrewbogott>	 but maybe that was a specific case, not to do with kernel upgrades
[21:54:20] <YuviPanda>	 andrewbogott: oh yeah, but that was them moving distro
[21:54:23] <YuviPanda>	 that won't work
[21:54:25] <YuviPanda>	 kernel itself is fine
[21:54:37] <andrewbogott>	 so what is it about the distro then, I wonder?
[21:54:54] <YuviPanda>	 good question. I don't know
[22:02:43] <andrewbogott>	 YuviPanda, anything I can do and/or document before I leave you to the holiday fates?
[22:03:31] <valhallasw`cloud>	 YuviPanda: I'm off to bed, I'll start working on a puppet manifest for sni vhosts tomorrow
[22:07:28] <valhallasw`cloud>	 YuviPanda: but the short version is 'https://www.digitalocean.com/community/tutorials/how-to-set-up-multiple-ssl-certificates-on-one-ip-with-nginx-on-ubuntu-12-04 works immediately'
[22:08:06] <HakanIST>	 hello
[22:09:57] <wikibugs>	 6Labs, 10Labs-Infrastructure: Start pdns after opendj - https://phabricator.wikimedia.org/T65717#1902290 (10Andrew) I don't know :(
[22:21:22] <wikibugs>	 6Labs, 10Labs-Infrastructure: Start pdns after opendj - https://phabricator.wikimedia.org/T65717#1902327 (10Andrew) ...Actually, yes, it probably still is the case.  Since most likely the issue was pdns getting upset at a service interruption, which would still happen with an openldap outage.  On the other han...
[22:21:58] <wikibugs>	 6Labs, 10Labs-Infrastructure: Start pdns after opendj - https://phabricator.wikimedia.org/T65717#1902341 (10Andrew) The good news is that shutting off that ldap-backed pdns server is part of the quarterly goal for next quarter.  So we only have to ignore this bug for a few more months.
[22:45:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-worker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[22:46:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-worker-1002 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]
[22:47:05] <YuviPanda>	 andrewbogott: any idea why some instances are still stillborn?
[22:47:24] <YuviPanda>	 andrewbogott: worker-1003 is stillborn for example
[22:47:30] <YuviPanda>	 andrewbogott: outside of that it's all good I think
[23:32:07] <travis-ci>	 wikimedia/nagf#30 (docker - 2961b2c: YuviPanda) The build passed. - https://travis-ci.org/wikimedia/nagf/builds/98605164
[23:33:32] <travis-ci>	 wikimedia/nagf#31 (docker - 75e62bd: YuviPanda) The build passed. - https://travis-ci.org/wikimedia/nagf/builds/98605454
[23:38:41] <wikibugs>	 6Labs: Create new labs project for ifttt - https://phabricator.wikimedia.org/T122378#1902689 (10madhuvishy) 3NEW a:3yuvipanda
[23:41:25] <wikibugs>	 6Labs: Create new labs project for ifttt - https://phabricator.wikimedia.org/T122378#1902719 (10madhuvishy)
[23:41:30] <wikibugs>	 6Labs, 7Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#1902718 (10madhuvishy)