[01:40:27] Coren: even I added 'enable-threads = true' in uwsgi.ini after [uwsgi], and it says 'python threads support enabled', no threads getting started. [02:00:07] 10Tool-Labs: Block xovibot user-agent - https://phabricator.wikimedia.org/T90636#1076075 (10scfc) I don't see that bot having read `robots.txt` but the requests seem to have ceased: ``` root@tools-webproxy:~# fgrep -c XoviBot /var/log/nginx/access.log*; ls -l /var/log/nginx/access.log* /var/log/nginx/access.log... [02:00:23] 10Tool-Labs: Block xovibot user-agent - https://phabricator.wikimedia.org/T90636#1076076 (10scfc) 5Open>3Resolved [02:16:36] PROBLEM - SSH on tools-exec-catscan is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:27] RECOVERY - SSH on tools-exec-catscan is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 (protocol 2.0) [03:13:00] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:14:52] PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:28:01] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [03:39:33] labs-morebots: still here? [03:39:33] I am a logbot running on tools-exec-13. [03:39:33] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [03:39:33] To log a message, type !log . [04:43:07] 10Tool-Labs, 7Tracking: [tracking] Block spider / web crawler on tool labs - https://phabricator.wikimedia.org/T70300#1076173 (10scfc) 5Open>3Invalid A tracking task with only one (resolved) blocking task doesn't make much sense to me :-). There are certainly some things to improve about our spider treatm... [05:10:53] PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [43200.0] [05:14:40] PROBLEM - Puppet staleness on tools-webgrid-07 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [43200.0] [06:28:17] 6Labs, 10Wikimedia-Labs-wikitech-interface: Wikitech doesn't allow to associate a hostname with a public ip address - https://phabricator.wikimedia.org/T90856#1076195 (10Andrew) 5Open>3Resolved a:3Andrew Must've been in a 24-hour cache entry... it's working correctly now. [06:44:22] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:09:22] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [11:12:04] 6Labs, 10Wikimedia-Labs-wikitech-interface: Wikitech doesn't allow to associate a hostname with a public ip address - https://phabricator.wikimedia.org/T90856#1076338 (10Kelson) I confirm it works now. THX. [14:39:44] RECOVERY - Puppet staleness on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:45:10] 6Labs, 10Tool-Labs: Investigate OOMs in trusty webgrid nodes - https://phabricator.wikimedia.org/T91194#1076429 (10yuvipanda) 3NEW [14:45:34] 6Labs, 10Tool-Labs: Investigate OOMs in trusty webgrid nodes - https://phabricator.wikimedia.org/T91194#1076436 (10yuvipanda) I have created and set up tools-webgrid-07 to relieve pressure in the meantime. [14:47:13] 10Wikimedia-Labs-Infrastructure: Move LabsDB aliases and NAT to DNS and LabsDB servers - https://phabricator.wikimedia.org/T63897#1076437 (10yuvipanda) /me pokes @Coren [15:07:57] hi, how can I make python work as cgi? [15:09:12] pirsquared: one option is https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Python_.28uwsgi.29 [15:10:50] so hard to make it work as cgi compared to php... [15:11:28] !log tools pooled in tools-webgrid-07 to lighty webgrid, moving some tools off -05 and -06 to relieve pressure [15:11:34] Logged the message, Master [15:11:47] pirsquared: you shouldn’t be using CGI anyway. the link I mentioned is the wsgi support in python that we have. [15:11:53] and works by default well with flask [15:12:00] what's the difference? [15:12:27] https://stackoverflow.com/questions/4929626/what-are-wsgi-and-cgi-in-plain-english [15:12:32] wsgi is also a lot faster than plain CGI [15:12:40] since it doesn’t spawn a new process for each request [15:13:06] for a ~50 line script is it overkill? [15:13:23] well, setting it up as CGI is harder than using wsgi [15:13:54] there is an example to set up fcgi with lighttpd, and lighttpd supports CGI if you wanna go that route, but it’s not one I’ll reccomend [15:14:10] besides, if you use flask (or a similar small framework) it’ll ‘just work’ for this wsgi deployment [15:47:24] YuviPanda|brb: does that work for python2 too? [15:55:54] pirsquared: yes python2 is default [16:10:56] YuviPanda|brb: now I can get it to log to uwsgi.log, but it just gives "Internal error" and there's no error [16:25:56] pirsquared: I'm eating now can help debug in a bit. Which tool is this? [16:26:09] I just tried starting a new one [16:26:18] /data/project/usergraph [16:33:51] I think I'll just use PHP, meh [17:12:06] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [17:12:25] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:15:23] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [17:30:22] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:11] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:31] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [18:37:18] PROBLEM - Puppet failure on tools-uwsgi-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:42] 10Wikibugs: Wikibugs should report the Priority if it is set during task creation - https://phabricator.wikimedia.org/T91202#1076582 (10Quiddity) 3NEW [18:42:40] pirsquared: sorry, I can take a look and try to help tomorrow [18:44:53] Hi all. I need some help setting up BigBrother on toollabs [18:46:07] I put "webservice" in a .bigbrotherrc file in the tool's home [18:46:15] How would I know if this worked? [18:56:18] Jarry1250: stop the webservice, it should restart automatically after some time [18:56:37] sitic: good idea [18:57:10] sitic: Any idea how long that takes? [18:57:18] ...immediate [18:57:32] just got emailed [18:58:05] oh wow, didn't know it was so quick [18:58:47] maybe just lucky I suppose [18:59:26] No, really just that quick [19:47:18] 10Tool-Labs-tools-Other: Move Gerrit reports crontabs to Tools - https://phabricator.wikimedia.org/T88384#1076676 (10Nemo_bis) 5Open>3Resolved Well, crontabs added. [21:21:24] Bigbrother checks at least once a minute to make sure things are up but be aware that it will refuse to start any one job more than thrice in a 24h window. [21:22:33] (It'll email you to tell you that if it happens though, so you can take steps to restart it manually) [21:54:13] 6Labs, 10Wikimedia-Labs-wikitech-interface: wikitech is unresponsive/unable to log in - https://phabricator.wikimedia.org/T91218#1076771 (10scfc) 3NEW [22:08:30] Er, I went to comment on that phab ticket but phab-ldap seems to be down too [22:27:13] Jarry1250: Didn't you connect your Wikipedia account? [22:27:52] multichill: Probably. Logging in via OAuth doesn't work either. [22:28:03] Basically, I can't log into phab, that's what I'm saying :) [22:31:21] I can log in fine [22:31:41] (into phabricator) [22:36:58] I get a plain Wikimedia Error [22:37:10] (503 Service Unavailable) [22:39:04] Request: POST http://phabricator.wikimedia.org/auth/login/ldap:self/, from 10.64.0.171 via cp1043 cp1043 ([10.64.0.171]:80), Varnish XID 196772970 [22:39:23] Probably want to post that to -operations?