[04:21:30] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [05:00:43] 6Labs, 10Tool-Labs: Add tool-labs admins to `relic` project - https://phabricator.wikimedia.org/T118375#1800309 (10yuvipanda) I don't think there are privacy concerns since y'all already have NDAs. [05:01:24] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [05:04:39] !log toolserver-legacy add valhallasw and scfc to the project as admins, per https://phabricator.wikimedia.org/T118375 [05:04:58] 6Labs, 10Tool-Labs: Add tool-labs admins to `relic` project - https://phabricator.wikimedia.org/T118375#1800312 (10yuvipanda) 5Open>3Resolved a:3yuvipanda It's actually the toolserver-legacy project and I've added you both. [05:05:09] 6Labs, 10Tool-Labs: Add tool-labs admins to `toolserver-legacy` project - https://phabricator.wikimedia.org/T118375#1800315 (10yuvipanda) [07:26:40] RECOVERY - Puppet staleness on tools-k8s-bastion-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [07:51:16] YuviPanda: yaaaaay [07:51:30] inb4 kikibugs [07:51:33] haha [07:51:36] that's a good idea actually [07:51:39] legoktm: do you wanna do it on friday? [07:51:52] sure [07:52:05] legoktm: kk [07:58:27] legoktm: is it all py3 or py2? [07:58:30] py3 right? [07:58:35] all py3 [07:58:41] legoktm: the webservice? :P [07:58:48] oh, that's PHP [07:59:06] the webservice just git pulls, right? [07:59:08] it's like 5 lines, we can port it to whatever language [07:59:11] yeah [07:59:24] yeah can you port that to python too? [07:59:30] flask? [07:59:31] actually [07:59:34] no just let it be php [07:59:42] we shall port if that ends up being too complex [07:59:48] ok [07:59:49] legoktm: php now has a built in webserver doesn't it? [07:59:54] legoktm: and I guess that'll be enough for this [07:59:57] for debugging only [08:00:04] not for actual prod usage [08:00:11] 'for this' :P [08:00:30] I'd rather not rely on it :/ [08:00:41] I don't want to setup an entire lighttpd setup inside just for this [08:01:01] like, if it were flask I'd be ok with just running it with app.run() [08:01:10] instead of uwsgi [08:01:28] Warning [08:01:28] This web server was designed to aid application development. It may also be useful for testing purposes or for application demonstrations that are run in controlled environments. It is not intended to be a full-featured web server. It should not be used on a public network. [08:01:32] (https://secure.php.net/manual/en/features.commandline.webserver.php) [08:01:46] fiiiine [08:01:48] it's going to be like 10 lines of flask [08:01:52] and uwsgi [08:02:00] I'm trying to avoid the extra actual webserver [08:02:18] oh well, let it be php, we'll figure out something with php-fcgi [08:02:23] err [08:02:25] php-fpm [08:02:48] I should port nagf to just php-fpm first actually [08:02:51] and then can reuse that [08:09:01] hmmmmmmm [08:09:05] that requires running two processes [08:09:08] php-fpm and nginx >_> [08:09:12] <_< [08:09:24] legoktm: wanna rewrite it in node? that has a built in webserver! :D [08:09:44] so does python! [08:09:58] legoktm: I don't think you're supposed to use that on a public network :P [08:10:19] lol [08:10:23] well, flask it is then [08:10:31] kk [08:11:06] YuviPanda: tell me to get off my keyboard and go to sleep [08:11:25] greg-g: please get off your keyboard and go to sleep. NOW [08:11:34] good idea. [08:34:50] 6Labs, 10Tool-Labs: Define base Wikimedia Docker container - https://phabricator.wikimedia.org/T118446#1800488 (10yuvipanda) 3NEW [09:57:29] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1800525 (10valhallasw) Yes. Lighttpd buffers between the CGI app and the client, which allows the CGI app to finish earlier (because the limit is the pipe to lighttpd rather than the internet connection to the client). This th... [10:19:41] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:59:44] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [12:13:44] 6Labs: bigbrother trying to run and restart non-existing task (wikisource-bot) - https://phabricator.wikimedia.org/T118387#1800796 (10Billinghurst) @scfc same problem again today 2015-11-12 11:42:26 warn: job '' failed to start and betweemn 40-50 this batch [13:07:33] Hello, does anyone know whether a tool exists to see how many times and on what pages something (e.g. a book title) has been cited on wikipedia? [14:20:40] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:55:48] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:46] Coren: I'm working on Earwig's Copyvios detector. Do you know how I would get an API key? [15:24:00] ^ for the Yahoo BOSS API? [15:24:23] fhocutt: I... don't know who holds the credentials to it; but I know who to ask. Lemme find that out for you. [15:24:44] thanks! [15:27:53] 6Labs, 10Tool-Labs, 7Easy: Log CommitLimit to graphite - https://phabricator.wikimedia.org/T118465#1800985 (10valhallasw) 3NEW [15:31:15] 6Labs, 10Tool-Labs, 7Easy: Log CommitLimit to graphite - https://phabricator.wikimedia.org/T118465#1800998 (10valhallasw) In the case of Tool Labs, this will typically be equal to to (0.95*memory.MemTotal + memory.SwapTotal), so to plot the available commit space, one could plot (0.95*$1.memory.MemTotal + $... [15:31:34] 6Labs, 10Tool-Labs, 7Easy: Log CommitLimit to graphite - https://phabricator.wikimedia.org/T118465#1800999 (10valhallasw) [18:02:18] 6Labs, 10Tool-Labs, 5Patch-For-Review: Remove dependency of toollabs::checker on toollabs::submit and shut down bigbrother on tools-checker-01/tools-checker-02 - https://phabricator.wikimedia.org/T113744#1801344 (10scfc) 5Resolved>3Open I can't read my own handwriting: > This change instead depends tool... [18:07:56] Coren, any word on the API key setup? [18:08:05] 6Labs, 10Tool-Labs, 5Patch-For-Review: Remove dependency of toollabs::checker on toollabs::submit and shut down bigbrother on tools-checker-01/tools-checker-02 - https://phabricator.wikimedia.org/T113744#1801377 (10scfc) 5Open>3Resolved On `tools-checker-01` and `tools-checker-02`, I deleted `/etc/init/b... [18:11:27] 6Labs: bigbrother trying to run and restart non-existing task (wikisource-bot) - https://phabricator.wikimedia.org/T118387#1801397 (10scfc) @Billinghurst, I'm sorry, I had closed another task (T113744) prematurely and thought that there was only one `bigbrother` running (which I restarted), while there were thre... [18:13:32] fhocutt: Email sent; I expect an answer sometime today PST [18:13:37] all right, thanks! [18:13:45] I appreciate it. [18:57:42] Coren: do you remember why overcommit is disabled on exec hosts? [18:58:31] as SGE is supposed to keep memory usage in check already [19:42:10] YuviPanda: http://jvns.ca/blog/2015/11/09/docker-is-amazing/ [19:46:37] PROBLEM - Host tools-andrew-puppettest is DOWN: CRITICAL - Host Unreachable (10.68.21.109) [20:50:09] !log ores Deployed 334b4b2 with ores:b745570, revscoring:, editquality:b41b7c1, wikiclass:bbfa9ce, and wb-vandalism:a09d026 [20:50:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [20:52:26] valhallasw`cloud: that's both great and slightly terrible :D but mostly great [22:29:18] PROBLEM - Puppet failure on tools-redis-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:44:55] PROBLEM - Puppet failure on tools-redis-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:25:59] 6Labs, 3Labs-Sprint-104, 3Labs-Sprint-105: Learn about/document NFS and LVM setup - https://phabricator.wikimedia.org/T104220#1802542 (10Andrew) 5Open>3Resolved I think this is as done as it's going to be. [23:26:13] 6Labs, 3Labs-Sprint-109, 3labs-sprint-110: Make a menu of potential new labs features, invite comments from users - https://phabricator.wikimedia.org/T101769#1802545 (10Andrew) a:5Andrew>3None [23:26:29] 10MediaWiki-extensions-OpenStackManager: OpenStackManager 'configure instance' clears any settings not visible in the configure GUI. - https://phabricator.wikimedia.org/T94851#1802547 (10Andrew) a:5Andrew>3None [23:26:47] 6Labs: Need Horizon dashboard for manipulating service groups - https://phabricator.wikimedia.org/T91989#1802548 (10Andrew) a:5Andrew>3None [23:26:54] 6Labs: New Horizon dashboard for user management - https://phabricator.wikimedia.org/T91988#1802552 (10Andrew) a:5Andrew>3None [23:27:00] 6Labs: Sort out labs user privs in Horizon vs. Wikitech - https://phabricator.wikimedia.org/T91830#1802553 (10Andrew) a:5Andrew>3None [23:44:59] 6Labs, 3Labs-Sprint-109, 7Monitoring, 5Patch-For-Review, and 3 others: Monitor nova services - https://phabricator.wikimedia.org/T90784#1802597 (10Andrew) a:5Andrew>3None [23:46:03] 6Labs, 10Labs-Infrastructure, 3labs-sprint-117: switch to keystone api v3 - https://phabricator.wikimedia.org/T115027#1802598 (10Andrew) a:5Andrew>3None [23:46:20] 6Labs: Two small instances: for WikiToLearn development - https://phabricator.wikimedia.org/T115282#1802599 (10Andrew) a:5Andrew>3None [23:46:36] 6Labs, 10Labs-Infrastructure: Precise instances say "ImportError: No module named cc_power_state_change" on startup - https://phabricator.wikimedia.org/T103808#1802600 (10Andrew) a:5Andrew>3None