[06:57:58] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:59:48] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:24:50] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:28:02] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [08:49:10] Could any admin restart xtools's web service? [08:51:06] Zhaofeng_Li: doing [08:52:17] Zhaofeng_Li: done [08:52:36] YuviPanda: Many thanks! [08:52:43] yw [11:11:27] 3Wikimedia-Labs-General: Create ping.wmflabs.org - https://phabricator.wikimedia.org/T56427#806452 (10Qgil) [11:39:34] 3Wikimedia-Labs-General: DNS problem with instance on Wikimedia Labs (apparently two instances with same name) - https://phabricator.wikimedia.org/T73595#806569 (10Prtksxna) [11:56:05] Greetings! [11:56:37] I'm having some issues with my flask app's fcgi config [11:56:52] Log: http://pastebin.com/pcLyDXSZ [11:57:34] > If you're trying to run your app as a FastCGI backend, make sure you're using the FastCGI-enabled version. [12:00:08] valhallasw`cloud, ping [12:05:19] planemad: huh, that's weird [12:05:22] a PHP error? :| [12:05:35] anyway, try running /data/project/wikiatlas2014/wikimapsatlas-server/api/app.fcgi directly [12:05:40] you might need to chmod 755 it [12:06:08] i've already set the permissions. how do i run it? [12:06:17] just run it from bash? [12:06:31] i.e. type /data/project/wikiatlas2014/wikimapsatlas-server/api/app.fcgi in a shell [12:06:37] and see if you get any errors [12:06:40] btw this is my lightpd.conf: https://github.com/WikimapsAtlas/wikimapsatlas-server/blob/master/api/lighttpd.conf [12:07:14] you probably want ( "^/wikiatlas2014/api$" => "/wikiatlas2014/api/" ) instead of the current redirect, but looks good otherwise [12:08:06] ok [12:08:10] ah -bash: /data/project/wikiatlas2014/wikimapsatlas-server/api/app.fcgi: /wikimapsatlas-server/api/bin/python: bad interpreter: No such file or directory [12:10:12] valhallasw`cloud, guess it was the incorrect path to python, will fix and try again [12:10:15] ok! [12:25:15] 3Wikimedia-Fundraising, Tool-Labs-tools-Other: grrrit-wm: runs into infinite loop if unable to join channels - https://phabricator.wikimedia.org/T74523#806799 (10Lydia_Pintscher) [12:34:30] valhallasw`cloud, i've got the app running now, but it seems that the urls are getting redirected to the public_html folder [12:34:40] http://pastebin.com/5uxBEX7c [12:35:29] we already have our website running at http://tools.wmflabs.org/wikiatlas2014/ [12:36:04] now am not sure how the api urls should be configured [12:36:37] 3Wikimedia-Labs-General: setup labs project for continuous integration jobs - https://phabricator.wikimedia.org/T55978#806843 (10zeljkofilipin) [12:38:25] hmm, I have a feeling I know how to solve this, will be back if I can't [12:38:29] thanks [13:07:57] planemad: if you can read it, check /data/project/gerrit-patch-uploader/error.log for what it should look like [13:14:45] https://www.irccloud.com/pastebin/wcOkTvVb [13:14:48] planemad: ^^ [13:15:15] planemad: so the fcgi handler is not picking up the request [13:15:49] valhallasw`cloud, got it sorted out, thanks :) [13:15:59] *thumbsup* [13:17:49] this is essentially the first time in life i'm hitting the backend. so good progress overall :D [13:18:22] planemad: yay :) [13:18:42] now I know a little bit more about the mysteries of the universe [13:21:44] YuviPanda, also your advice on importing things the right way is making more sense. Suffering major python bites.. [13:22:06] planemad: :) 'explicit better than implicit' :) [13:22:48] planemad: reccomend opening a python shell, and typing 'import this' for poetry [13:23:00] YuviPanda: weren't you working on a 'instant flask project, just add water' for tools? [13:23:25] valhallasw`cloud: yeah, uwsgi for tools. [13:23:26] YuviPanda, :O [13:23:58] planemad: :) [13:24:11] valhallasw`cloud: let's just say I've been distracted, mostly by monitoring and other opsy stuff [13:24:17] valhallasw`cloud: haven't really worked on toollabs in a while [13:24:20] :) [13:24:52] valhallasw`cloud: but I'm starting on understanding our labsdb setup, so that should help fix the last remaining class of big toollabs problems :) [13:25:02] coolcoolcool [14:01:39] its amazing how the Special:NovaInstance auto-corrects after you logout and login in wikitech [14:02:23] tonythomas: heh [14:02:29] by amazing you mean terrible [14:02:50] of course - but happy once we get the things back, I would say [14:05:48] :) [17:03:45] YuviPanda: I'm going to be gone a lot today (Dr. appointment, nothing serious.) I think the best way forward with wikitech-test is to rm the openstack databases and start fresh -- I'm getting crazy internal python errors that can't possible be the result of broken config. [17:03:53] So, I'll work on that in what corners of the day I have today. [17:04:01] andrewbogott: on a fresh instance? [17:04:03] Sorry this is holding you up for so long. [17:04:10] andrewbogott: 'tis ok, I'll find other things for now :) [17:04:15] I think I'll try to keep the instance and just rm the database dir. [17:04:25] ok! [17:04:26] So that ldap sticks around, since it seems to be working. [17:16:47] Coren: do you think you'll have time today to fix bigbrother for trusty? [17:17:11] Yes, I'm stalled on one thing on my todo because network, that'll free some time for me. [17:17:46] Coren: yay, cool :) [17:18:06] I should either learn perl or rewrite it in python. Think we should make all significantly-large scripts in our repo be python [17:20:22] Either way, learning perl would be Good For You™, rewrite or no. [17:22:11] Coren: :) [17:25:41] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:44:15] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#807530 (10Awjrichards) [18:01:28] YuviPanda: https://gerrit.wikimedia.org/r/#/c/177262/1 [18:01:35] looking [18:02:09] That supports 'webservice [--release foo] lighttpd|tomcat' in .bigbrotherrc [18:03:04] Old syntax 'webservice (-lighttpd|-tomcat)' still uses webservice so won't break anything. [18:05:52] Coren: does the $2 to --release=trusty? [18:05:56] so much regex [18:06:19] if so that sounds good [18:27:35] It won't take an =, but that's because that's not what --help sayz. :-) [18:27:48] usage: webservice2 [-h] [--release {precise,trusty}] [18:31:47] Coren: oh, right. [18:31:50] Coren: that works too. [18:32:03] wonders of not having to parse commandline arguments by hand :) [18:32:09] it'll also take -r precise, for example [18:35:30] Coren: but if it does, then sure, wfm :) [19:00:06] anyone know anything about tools.wmflabs.org/enwp10/cgi-bin/pindex.fcgi ? [19:06:44] (03CR) 10Legoktm: [C: 032] Rewrite some flow-style YAML [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/176989 (owner: 10Awight) [19:06:49] (03Merged) 10jenkins-bot: Rewrite some flow-style YAML [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/176989 (owner: 10Awight) [19:07:51] !log tools.lolrrit-wm restarting grrrit-wm for https://gerrit.wikimedia.org/r/176989 [19:07:56] Logged the message, Master [19:09:35] (03CR) 10Legoktm: "Test" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/176989 (owner: 10Awight) [20:31:30] 3Wikimedia-Labs-General: DNS problem with instance on Wikimedia Labs (apparently two instances with same name) - https://phabricator.wikimedia.org/T73595#808396 (10Andrew) 5Open>3Resolved [21:20:00] !log wikidata-build deleted all instances. wikidata-jenkins* were removed from the jenins config thus stoped receiving jobs. wikidata-build* was not used anymore. project can be deleted. [21:20:03] Logged the message, Master [21:30:07] Coren: ping [21:31:00] QueenOfFrance: Hello? [21:31:46] Coren: hi, not sure if you're the right person to poke. One of petan's irc bots seems to be running into excess flood issues, and I don't think petan's around (wm-bot4) [21:33:12] (03PS1) 10Mattflaschen: Rename #wikimedia-corefeatures to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/177371 [21:33:37] !log deployment-prep updated OCG to version 08e94b19c3f17e699d7e53d9605f65c58e17ea0e [21:33:40] Logged the message, Master [21:34:53] (03PS1) 10Mattflaschen: Rename #wikimedia-corefeatures to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/177372 [21:35:15] QueenOfFrance: I'm actually a pretty poor choice; that bot lives on its own project and I've nothing to do with it. :-( andrewbogott might be able to help (but he's not around), and so could Addshore and Madman that I know of who are involved with it. [21:35:25] (03CR) 10EBernhardson: [C: 031] Rename #wikimedia-corefeatures to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/177371 (owner: 10Mattflaschen) [21:35:33] THO also, if I remember right. [21:35:41] Coren: see, you were a good choice because you know who should have been poked instead of you :) [21:35:59] (03CR) 10EBernhardson: [C: 031] Rename #wikimedia-corefeatures to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/177372 (owner: 10Mattflaschen) [21:36:24] Heh. [21:37:31] (03CR) 10Mattflaschen: [C: 04-1] "Self -1 until we get the board renamed, at which point I'll amend this." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/177372 (owner: 10Mattflaschen) [21:41:08] (03PS1) 10Awight: Add more repos maintained by Fundraising Tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/177378 [21:53:42] (03PS2) 10Mattflaschen: Rename #wikimedia-corefeatures to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/177372 [22:16:44] 3Tool-Labs-tools-Other: grrrit-wm: runs into infinite loop if unable to join channels - https://phabricator.wikimedia.org/T74523#808626 (10atgo) [22:22:28] 3Wikimedia-Labs-wikistats: Fix all the Wikia stats - https://phabricator.wikimedia.org/T61943#808638 (10Dzahn) hmm.. the cron job should be there but still no updates, i gotta check this [23:35:52] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]