[00:02:40] 3operations, Labs-Team: Puppet broken everywhere on labs - https://phabricator.wikimedia.org/T85293#943467 (10yuvipanda) 3NEW [00:03:05] 3operations, Labs-Team: Puppet broken everywhere on labs - https://phabricator.wikimedia.org/T85293#943467 (10yuvipanda) p:5Triage>3Unbreak! [00:04:20] YuviPanda: any hints where you did the oauth in quarry - I'm reading it on git hub [00:04:36] OrenBochman1: heya! I’m using a library for that, I think [00:04:53] hey [00:05:05] https://pypi.python.org/pypi/mwoauth/0.2.0 I believe [00:06:03] 3operations, Labs-Team: Puppet broken everywhere on labs - https://phabricator.wikimedia.org/T85293#943477 (10yuvipanda) Doesn't affect instances that have a non `virt1000` puppetmaster - self puppetmasters with latest ops/puppet clone and deployment-prep instances both work fine. [00:09:01] btw did you write up anything about connecting to the db via sh forwarding ? [00:09:48] OrenBochman1: I haven’t. Can you file a task in phab for that and assign it to me? [00:09:57] so I won’t forget next time :) [00:10:14] I;ll try [00:22:49] 3operations, Labs-Team: Puppet broken everywhere on labs - https://phabricator.wikimedia.org/T85293#943480 (10yuvipanda) Turns out it was a stray line in `/etc/puppet/auth.conf` in virt1000. Offending lines were: ```path /certificate_request auth any method find, save allow * method find, save``` The duplicate... [00:23:06] 3operations, Labs-Team: Puppet broken everywhere on labs - https://phabricator.wikimedia.org/T85293#943481 (10yuvipanda) @Joe think I can remove that file now? [00:24:56] 3operations, Labs-Team: Puppet broken everywhere on labs - https://phabricator.wikimedia.org/T85293#943489 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Things are recovering now. [00:34:07] Might be related to the fact that the instance is said to be still "building" [00:52:50] !log shinken built and set up python3-ldap package for shinkengen [00:52:54] Logged the message, Master [00:57:59] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:13] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:15] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:25] RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:31] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:33] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:45] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:49] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:49] RECOVERY - Puppet failure on tools-uwsgi-01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:50] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:57] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [00:59:24] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [00:59:32] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [00:59:58] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [00:59:59] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:06] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:15] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:15] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:43] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:47] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:54] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:54] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:58] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:00] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:04] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:10] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:16] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:18] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:46] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:50] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:56] RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:56] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:16] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:20] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:38] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:50] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [01:14:56] terrrydactyl: terrrydactyl: so, I kind of fixed the problem - [01:15:10] https://phabricator.wikimedia.org/T72199 has more [01:15:11] did you manage to ssh? [01:15:29] yeah [01:15:45] stuff on my end + some kind of overload? with tools-login so I used tools-dev instead [01:16:23] terrrydactyl: have you ever dealt with Flask? [01:16:38] yep! wikimetrics was done exclusively in flask [01:16:39] :) [01:16:45] oh right! [01:16:54] so https://tools.wmflabs.org/ links to some index.html pages [01:16:59] i see the websde up now [01:17:02] website [01:17:14] for missing-from-wikipedia it goes to https://tools.wmflabs.org/missing-from-wikipedia/ which I assume is an index.html page somewhere [01:17:34] either in public_html/ or in public_html/missing-from-wikipedia/ [01:17:37] I don't know which [01:18:42] looks like it's just https://tools.wmflabs.org/missing-from-wikipedia/index [01:18:43] also I had to change the @app.route [01:18:53] as valhallasw`cloud said, "so what's happening is this. lighttpd sends the request via fcgi to flask. But your flask app only has a route for @app.route('/index', " [01:18:56] he continued: "I think the easiest fix would be to change @app.route('/index' to @app.route('/'" [01:18:58] so I did that [01:19:45] okay [01:19:46] so there's probably some cleverness that someone (you) could do to make this all work better, so that the page that the tools.wmflabs.org directory points to can also .... something [01:19:56] :) [01:20:03] i'll fork your code today and take a look! [01:20:09] * YuviPanda waves [01:20:18] Hi YuviPanda! [01:20:20] now that i can see what it should look like, it's probably easier to check it out [01:20:23] was nice to see you in Nov [01:20:23] hi sumanah! [01:20:27] terrrydactyl: cool [01:20:30] sumanah: was nice to see you too! [01:20:41] terrrydactyl: sumanah [01:20:45] :) :) [01:20:56] sumanah: if you changed your flask code, you need to execute a ‘webservice restart’ as your tool for it to take effect [01:20:58] thanks YuviPanda! ambigious happy holidays to you too! [01:21:07] YuviPanda: yeah, I did that -- thanks! [01:21:21] ah, cool :) [01:22:09] * sumanah is reading Yuletide Treasure (fanfic exchange) -- thousands of stories just went live [01:22:18] nice! [01:22:28] * YuviPanda is reading up on debian packaging [01:22:33] sumanah, as i work through this, i can try updating the readme so that it'll be easier for people to figure out how to install/run it [01:22:44] terrrydactyl: sounds good to me! [01:23:07] terrrydactyl: in theory, at least, people who want to run it locally can just use the command-line version and feed it textfiles [01:23:45] yeah, but i like to have a web local version too in case i want to make any visual changes like adding an about page or something like that [01:24:36] nod nod nod [01:24:45] also other people might want to mod it or what have you [01:27:50] yep! [02:38:04] 3Labs-Team, Wikimedia-Labs-wikitech-interface: Separate wikitech machine from labs nova controller - https://phabricator.wikimedia.org/T85297#943541 (10yuvipanda) 3NEW [03:02:53] !log extdist switched extdist.wmflabs to point to extdist3 instead of extdist2 [03:02:56] Logged the message, Master [06:55:55] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:57:08] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:20:59] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:09] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [11:28:57] 3Tool-Labs: Trusty instances do not show the motd banners - https://phabricator.wikimedia.org/T85307#943684 (10scfc) 3NEW [11:30:52] 3Tool-Labs: Trusty instances do not show the motd banners - https://phabricator.wikimedia.org/T85307#943688 (10scfc) [11:50:43] 3Tool-Labs: Trusty instances do not show the motd banners - https://phabricator.wikimedia.org/T85307#943690 (10scfc) [[https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/1282046|This]] bug seems to be related. Apparently, the configuration of pam_motd is different for console and sshd logins: ``` scfc@too... [12:50:24] 3Beta-Cluster, operations, Labs-Team: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#943783 (10yuvipanda) betalabs' mediawiki-* instances filled up again, and I've set them to be dumped on NFS https://wikitech.wikimedia.org/w/index.php?title=Hiera%3ADeployment-prep&diff=13927... [15:25:04] 3Wikimedia-Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#943931 (10Se4598) 3NEW [15:49:23] 3Tool-Labs: Gerrit Patch Uploader does not work - https://phabricator.wikimedia.org/T85319#943939 (10Fomafix) 3NEW [15:58:25] 3Tool-Labs: Gerrit Patch Uploader does not work - https://phabricator.wikimedia.org/T85319#943952 (10jayvdb) [16:07:47] 3Tool-Labs: Gerrit Patch Uploader does not work - https://phabricator.wikimedia.org/T85319#943955 (10valhallasw) In which step, exactly? Yuvi, could this have something to do with uwsgi? I might have time for this in the weekend... [17:47:45] could anyone provide me a link to the gamification thing in labs? [18:43:05] 3Tool-Labs: Gerrit Patch Uploader does not work - https://phabricator.wikimedia.org/T85319#944031 (10Fomafix) I don't know if the problem is directly in the Gerrit Patch Uploader. After submit an empty HTML page is shown. Maybe related to this problem: The JavaScript optimized chooser does not work. It tries to... [19:01:51] valhallasw`cloud: hmm, uwsgi.log doesn’t show anything... [19:02:40] 3Wikimedia-Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#944034 (10yuvipanda) Neither icinga nor ganglia will be used in labs anymore. For graphs, use graphite.wmflabs.org, along with some help from tools.wmflabs.org/nagf Icinga ha... [19:02:51] 3Wikimedia-Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#944036 (10yuvipanda) 5Open>3declined a:3yuvipanda [19:10:45] 3Wikimedia-Labs-General: shinken.wmflabs.org redirects on https-login to http - https://phabricator.wikimedia.org/T85326#944041 (10Se4598) 3NEW [19:11:26] 3Wikimedia-Labs-General, Labs-Team: shinken.wmflabs.org redirects on https-login to http - https://phabricator.wikimedia.org/T85326#944047 (10yuvipanda) [19:15:54] 3Wikimedia-Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#944049 (10Se4598) maybe add a small html message instead or setup redirects? especially as the server are answering requests to that adress and the domain is not dead itself. O... [19:17:35] 3Wikimedia-Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#944050 (10yuvipanda) I'll try to find whoever set those services up initially and figure out what to do. [19:36:55] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:01:56] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [23:20:09] 3Tool-Labs: Add support for ASP.NET on Tool Labs - https://phabricator.wikimedia.org/T85142#944122 (10yuvipanda) 5Open>3Resolved a:3yuvipanda