[00:31:41] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [00:44:50] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2845882 (10scfc) 05Open>03declined Thanks. In that case, as there is no significant advantage in disabling `xdebug` and currently it is enabled (T72313), I'm opting to keeping the status quo. [01:06:11] 10Tool-Labs-tools-Other, 07Epic: Toolserver.org tools that have not been migrated (tracking) - https://phabricator.wikimedia.org/T60865#2845915 (10TTO) [01:06:13] 10Tool-Labs-tools-Other: Migrate http://toolserver.org/~purodha/sample/dbswithuser.php to Tool Labs - https://phabricator.wikimedia.org/T63028#2845913 (10TTO) 05Open>03stalled Purodha is sadly no longer with us. Does anyone know what this tool did or whether it is still required? If not I propose to close th... [01:08:16] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: Get access to an old database on tools-db - https://phabricator.wikimedia.org/T101709#2845916 (10TTO) 05Open>03Resolved a:03valhallasw No response, so I'm assuming this is resolved. [01:36:28] 06Labs, 10Tool-Labs: install php5-readline on bastion and exec hosts - https://phabricator.wikimedia.org/T136519#2845925 (10scfc) `php5-readline` is only available on Trusty, so I'll limit the patch to that. [02:53:25] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 1.46 ms [03:39:20] 10Tool-Labs-tools-Pageviews: Allow Langviews tool to track multiple articles, with reference to its Wikidata Q number - https://phabricator.wikimedia.org/T151888#2846006 (10MusikAnimal) @Wittylama Rereading this, I wasn't sure if you were aware that with Langviews, you can specify any Wikipedia project (or Wikiv... [03:49:06] 06Labs, 10Tool-Labs, 07Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#2846013 (10scfc) [03:49:09] 06Labs, 10Tool-Labs: Install debootstrap and fakechroot on tools - https://phabricator.wikimedia.org/T138138#2846008 (10scfc) 05Open>03Resolved a:05scfc>03chasemp Done by eb2d0069595b5e3c34f1d891d3845461cf6db22a. [04:19:40] 06Labs, 10Tool-Labs: DNS resolution sometimes fails on tools-bastion-03 - https://phabricator.wikimedia.org/T143194#2846046 (10Samwilson) Yep, seems to be. :-( I'm getting this, after quite a few requests: ``` PHP Fatal error: Uncaught exception 'GuzzleHttp\Exception\ConnectException' with message 'cURL erro... [05:00:09] 06Labs, 10Tool-Labs: BUB 503: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2846058 (10scfc) [05:00:12] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: jsub/jstart take 60 s due to /usr/local/bin/log-command-invocation CPU hunger - https://phabricator.wikimedia.org/T131700#2846060 (10scfc) [05:12:15] 06Labs, 10Tool-Labs: BUB 503: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2603370 (10scfc) The long invocation time for `jsub`, `webservice`, etc. comes IMHO as a side-effect of this task (or T147350 for the general case); example:... [05:12:56] 06Labs, 10Labs-Infrastructure: labservices1001 crashed and sent no pages - https://phabricator.wikimedia.org/T152368#2846069 (10Andrew) [05:15:36] 06Labs, 10Labs-Infrastructure, 10Monitoring: toolschecker fell to pieces when labs-ns0 went down - https://phabricator.wikimedia.org/T152369#2846085 (10Andrew) [05:18:25] 06Labs, 10Continuous-Integration-Infrastructure: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846099 (10Andrew) [06:05:03] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [06:23:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:58:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:00:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:24] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.95 ms [07:39:15] 06Labs, 10Labs-Infrastructure, 10Monitoring: labservices1001 crashed and sent no pages - https://phabricator.wikimedia.org/T152368#2846223 (10Peachey88) [07:48:24] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [08:09:41] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2846235 (10Marostegui) I have started transferring data from labsdb1010 to labsdb1011. Once we have both up we can try to tes... [08:45:40] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [08:50:18] 06Labs, 10Continuous-Integration-Infrastructure: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846271 (10hashar) I have looked at it / filled a task about it ages ago but can not find it anymore. The issue is the DHCP server on labs only yield a single DN... [09:00:15] 06Labs, 10Continuous-Integration-Infrastructure: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846279 (10hashar) Found it. T137460#2383979 and others have all the details. Namely the DHCP lease has: ``` option domain-name-servers 208.80.155.118; ``` And i... [09:00:32] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2368900 (10hashar) [09:00:35] 06Labs, 10Continuous-Integration-Infrastructure: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846285 (10hashar) [09:02:58] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org") - https://phabricator.wikimedia.org/T137460#2846286 (10ha... [09:11:03] 06Labs, 10Labs-Infrastructure, 10DBA: Create labsdb_accounts db on m5 to store state about labsdb accounts - https://phabricator.wikimedia.org/T152377#2846295 (10yuvipanda) [09:16:42] . [09:41:16] I followed instructions in https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy but cannot browse to Http://http://etytree-1.etytree.eqiad.wmflabs/ [09:41:30] sorry I meant http://etytree-1.etytree.eqiad.wmflabs/ [09:42:19] my ip address is 208.80.155.129 [09:42:28] any suggestion? [09:53:51] 06Labs, 10Labs-Infrastructure, 10DBA: Create labsdb_accounts db on m5 to store state about labsdb accounts - https://phabricator.wikimedia.org/T152377#2846397 (10jcrespo) a:05yuvipanda>03jcrespo [10:29:43] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 201.07 ms [10:38:46] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [10:40:52] (03PS1) 10Jcrespo: Add fake passwords for labspuppet and labsdbaccounts databases [labs/private] - 10https://gerrit.wikimedia.org/r/325274 (https://phabricator.wikimedia.org/T152377) [10:41:03] 10Tool-Labs-tools-Other: Migrate http://toolserver.org/~purodha/sample/dbswithuser.php to Tool Labs - https://phabricator.wikimedia.org/T63028#2846452 (10Nemo_bis) The tool took a username and list the wikis where an account with said username existed. Unlike the other similar tools, it also gave the local user_... [10:42:45] (03CR) 10Jcrespo: [C: 032 V: 032] Add fake passwords for labspuppet and labsdbaccounts databases [labs/private] - 10https://gerrit.wikimedia.org/r/325274 (https://phabricator.wikimedia.org/T152377) (owner: 10Jcrespo) [10:51:46] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2846466 (10Marostegui) labsdb1011 is up and running with a single channel. I will test two of them too [11:34:23] 10Tool-Labs-tools-Pageviews: Allow Langviews tool to track multiple articles, with reference to its Wikidata Q number - https://phabricator.wikimedia.org/T151888#2846540 (10Wittylama) @MusikAnimal My specific use-case for this request is based on this project: https://www.wikidata.org/wiki/Wikidata:Europeana_Art... [12:33:47] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2846647 (10jcrespo) ``` db1095$ check_private_data.py -- Non-public databases that are present: DROP DATABASE IF EXISTS `tes... [12:53:42] I followed instructions in https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy but cannot browse to http://etytree-1.etytree.eqiad.wmflabs/ [12:53:55] any suggestion? [12:57:52] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2846675 (10Marostegui) >>! In T152194#2846647, @jcrespo wrote: > ``` > db1095$ check_private_data.py > -- Non-public databas... [12:59:53] hey Epantaleo [13:00:05] hi Yuvi [13:00:21] Epantaleo: have you ever used a SOCKS proxy before? [13:00:35] some years ago [13:00:40] I see [13:01:01] I don't know who wrote those instructions, but if you don't care for it to remain private, I'd highly recommend using https://wikitech.wikimedia.org/wiki/Help:Proxy [13:01:03] much simpler [13:01:13] are you on Windows or? [13:01:20] macosx [13:01:41] ok thanks al ot [13:04:27] Epantaleo: yw [13:05:25] BTW, why this "foxyproxy" thing? [13:05:55] In Chromium I usually just go to settings and add the SOCKS proxy there (but maybe that doesn't support forwarding?) [13:06:48] no idea who wrote it, and I don't think it belongs in 'Access' [13:06:57] (ssh tunneling is simpler anyway) [13:10:48] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2846706 (10Marostegui) I have started transferring the data to labsdb1009. Also took a backup of the existing data, just in... [13:10:50] yep, I usually just do ssh -D [13:11:16] mostly because I keep forgetting any other method :) [13:16:03] ho [13:16:51] doing the set up; how do I know if I'm using MediaWiki-Vagrant? and therefore if I sould use port 80 or 8080? [13:19:07] Epantaleo: if you don't know you're using mediawiki-vagrant, you aren't [13:19:16] you should use whatever port number your server is listening on [14:04:13] 06Labs, 10Tool-Labs: About 71 users are missing replica.my.cnf - https://phabricator.wikimedia.org/T140592#2469834 (10chasemp) This should be resolved via T149933 where the mechanism in charge is being rewritten. This could be viewed as closing criteria for that work. [14:06:43] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841828 (10chasemp) >>! In T152212#2845634, @valhallasw wrote: > * Exceptions to this rule can be made on a case-by-case basis. Please contact us with y... [14:36:11] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#2846917 (10yuvipanda) [14:54:06] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2846967 (10Marostegui) labsdb1009 is now up and running. The three servers are replicating fine. I have also enabled SSL. R... [15:44:41] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2847231 (10jcrespo) > I have checked the current labs servers and they have set 25% RAM for the buffer pool size. That was b... [15:50:19] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2847244 (10chasemp) a:05yuvipanda>03None >>! In T149933#2843924, @... [15:59:20] I'm trying to reset my password, but the password reset mail is not being sent. Can you help? [16:03:21] SPF|Cloud: are you talking about on wikitech? [16:03:32] err yes, sorry for the confusion [16:03:39] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2847287 (10yuvipanda) a:03yuvipanda After more chat, we decided that... [16:04:03] what is teh account username and shell name? andrewbogott^ care to take a look? [16:04:14] 06Labs, 10Tool-Labs: About 71 users are missing replica.my.cnf - https://phabricator.wikimedia.org/T140592#2847294 (10scfc) [16:04:14] Both 'southparkfan' [16:04:16] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2847295 (10scfc) [16:05:01] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Create labsdb_accounts db on m5 to store state about labsdb accounts - https://phabricator.wikimedia.org/T152377#2847309 (10jcrespo) 05Open>03Resolved [16:05:03] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2847310 (10jcrespo) [16:05:16] SPF|Cloud: the email associated with that acount is @hotmail.com — is that really what you want? [16:05:29] yes, that is my email address [16:05:42] ok [16:06:30] Let me see if it works for me... [16:07:56] yep, works for me [16:08:09] SPF|Cloud: I got the email more-or-less immediately, subject line "Account details on Wikitech" [16:08:22] So I'm inclined to blame your spam filter, although thats an easy excuse... [16:08:32] let me look [16:09:29] it looks like Outlook is putting this mail in some other folder [16:10:41] in short, not a Wikitech failure.... just some filter annoying me. The password has been reset now. Sorry for the inconvenience [16:11:22] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2847356 (10jcrespo) As a reminder, ALTER TABLEs on m5, with just a few... [16:12:08] SPF|Cloud: no worries, glad you have it sorted [16:12:21] Many were confused when that happened on gmail, too [16:12:51] I suspect our emails never returned to the same rate of "success" as before [16:24:09] 06Labs, 10Recommendation-API: Request increased quota for recommendation-api labs project - https://phabricator.wikimedia.org/T152120#2839107 (10chasemp) +1 [16:24:38] 06Labs, 10Recommendation-API: Request increased quota for recommendation-api labs project - https://phabricator.wikimedia.org/T152120#2847404 (10Andrew) a:03Andrew [16:30:58] 06Labs: Request increased quota for etytree labs project - https://phabricator.wikimedia.org/T152417#2847419 (10Epantaleo) [16:37:28] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2847437 (10scfc) I just looked at the latest job that failed (`tail -10000 /var/lib/gridengine/default/common/accounting | fgrep tools.suggestbot`): ``` webgrid-lighttpd:tools-webgrid-lig... [16:50:55] !log tools Released floating IPs from decommissioned tools-exec-12[01-11] instances [16:50:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:53:28] !log tools Terminated deprecated instances: "tools-exec-1201", "tools-exec-1202", "tools-exec-1203", "tools-exec-1205", "tools-exec-1206", "tools-exec-1207", "tools-exec-1208", "tools-exec-1209", "tools-exec-1210", "tools-exec-1211" (T151980) [16:53:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:53:32] T151980: Reduce Precise OGE exec hosts to 10 - https://phabricator.wikimedia.org/T151980 [16:53:53] PROBLEM - Host tools-exec-1207 is DOWN: CRITICAL - Host Unreachable (10.68.17.113) [16:54:19] PROBLEM - Host tools-exec-1205 is DOWN: CRITICAL - Host Unreachable (10.68.17.91) [16:55:09] PROBLEM - Host tools-exec-1202 is DOWN: CRITICAL - Host Unreachable (10.68.16.57) [16:55:23] PROBLEM - Host tools-exec-1201 is DOWN: CRITICAL - Host Unreachable (10.68.17.49) [16:55:35] PROBLEM - Host tools-exec-1209 is DOWN: CRITICAL - Host Unreachable (10.68.17.129) [16:55:47] PROBLEM - Host tools-exec-1203 is DOWN: CRITICAL - Host Unreachable (10.68.16.133) [16:55:51] PROBLEM - Host tools-exec-1211 is DOWN: CRITICAL - Host Unreachable (10.68.17.64) [16:55:53] PROBLEM - Host tools-exec-1210 is DOWN: CRITICAL - Host Unreachable (10.68.17.147) [16:56:17] oh shinken-wm why? [16:56:34] YuviPanda: ^ what did I miss for shinken? [16:56:59] PROBLEM - Host tools-exec-1208 is DOWN: CRITICAL - Host Unreachable (10.68.16.151) [16:57:05] bd808: LDAP garbage cleaning only sometimes works, I think. andrewbogott was investigating this earlier [16:57:30] PROBLEM - Host tools-exec-1206 is DOWN: CRITICAL - Host Unreachable (10.68.17.105) [16:58:16] I don't think I ever did anything beyond complain about it :( [17:02:34] 06Labs, 10Recommendation-API: Request increased quota for recommendation-api labs project - https://phabricator.wikimedia.org/T152120#2847567 (10Andrew) 05Open>03Resolved [17:08:22] andrewbogott: oh :( [17:08:32] andrewbogott: bd808 I hope we don't end up with shinken complaining about it every day for 10 instances [17:08:44] I guess we should kill shinken, but prometheus is ofc not fully done. [17:08:45] boo [17:23:24] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2847702 (10scfc) It is indeed a problem with `webservicemonitor`. When it decides to restart the webservice for `tools.suggestbot`, it actually has `qstat`'s output as (inter alia): ```... [17:27:41] 06Labs: Request increased quota for etytree labs project - https://phabricator.wikimedia.org/T152417#2847419 (10bd808) Related to https://meta.wikimedia.org/wiki/Grants:IEG/A_graphical_and_interactive_etymology_dictionary_based_on_Wiktionary [18:02:12] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Reduce Precise OGE exec hosts to 10 - https://phabricator.wikimedia.org/T151980#2834080 (10scfc) (The #Shinken configuration gets regenerated on every Puppet run. So up to 30 minutes of false alarms are to be expected.) [18:31:22] 10Tool-Labs-tools-Pageviews: Allow Langviews tool to track multiple articles, with reference to its Wikidata Q number - https://phabricator.wikimedia.org/T151888#2848012 (10MusikAnimal) @Wittylama Got it. This is a bit of a specialized scenario, that I may not be able to support in a way that's consistent with o... [18:56:01] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Migrate existing labs users from the old servers, if possible using roles and start maintaining users on the new database servers, too - https://phabricator.wikimedia.org/T149933#2848149 (10jcrespo) I have added the several admin users to labsdb1009... [19:00:21] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Provision sanitized data on labsdb1009, labsdb1010, labsdb1011 with from db1095 - https://phabricator.wikimedia.org/T152194#2848170 (10jcrespo) I have added the 3 labsdb hosts to tendril, cleaned up its accounts, added the admin ones that labs host... [19:01:38] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2848179 (10scfc) … and it does treat it as "running", but `registered_webservices` does not contain `suggestbot` at those times. Hmmm. [19:03:41] 06Labs, 10Labs-Infrastructure, 06Operations: cronspam from labtestservices2001 /etc/dns-floating-ip-updater.py > /dev/null - https://phabricator.wikimedia.org/T152439#2848181 (10RobH) [19:31:23] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 13Patch-For-Review: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.... - https://phabricator.wikimedia.org/T137460#2848327 [19:49:43] 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: labservices1001 down - https://phabricator.wikimedia.org/T152340#2848411 (10fgiunchedi) [19:49:54] 06Labs, 10Labs-Infrastructure, 10Monitoring, 07Wikimedia-Incident: labservices1001 crashed and sent no pages - https://phabricator.wikimedia.org/T152368#2848412 (10fgiunchedi) [19:50:05] 06Labs, 10Labs-Infrastructure, 10Monitoring, 07Wikimedia-Incident: toolschecker fell to pieces when labs-ns0 went down - https://phabricator.wikimedia.org/T152369#2848413 (10fgiunchedi) [19:50:22] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2848414 [19:59:24] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2848466 (10scfc) I disabled `webservicemonitor` for `suggestbot` at 19:39Z, and at 19:46Z something deleted the proxy entry on `tools-proxy-01` (`/var/lib/redis/tools-proxy-01-6379.aof`):... [20:21:50] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2848557 (10scfc) `webservice start` → proxy entry created and web server running on `tools-webgrid-lighttpd-1416` → ``` Mon Dec 5 20:17:29 UTC 2016: *3 Mon Dec 5 20:17:29 UTC 2016: $4 M... [20:31:57] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841828 (10bd808) I don't think either of these projects qualifies as "unauthenticated" in that in my understanding OAuth is used to authenticated the r... [20:39:18] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: Use of uninitialized value in print at /usr/local/sbin/bigbrother line 210 - https://phabricator.wikimedia.org/T144955#2848643 (10bd808) 05Open>03Resolved The rewrite has been running for several days with no apparent errors. No log currently exist... [20:46:25] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2848664 (10chasemp) I take the term `unauthenticated` to mean end users are not directly authenticating to the resources under consumption, i.e. obfusca... [20:56:26] 06Labs, 10Quarry, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2848690 (10bd808) The [[https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools/Rules&diff=prev&oldid=120104|existing rule was introduced]]... [20:58:30] I cant connect via putty [20:58:42] to tool labs [21:01:05] Freddy2001: can you give us some more details? What error messages if any are you seeing? [21:01:27] and what hostname are you trying to connect to? [21:02:44] The error is: publickey, hostbased [21:02:56] i try connect to login.tools.wmflabs.org [21:03:30] ok. It sounds like your ssh key is being rejected. Let me see if I can find any more details in the server side logs [21:03:57] Freddy2001: has this worked for you in the past or is this the first time you are trying to conenct? [21:13:10] Freddy2001: I'm not seeing any recent ssh key failures on the login.tools server so maybe you haven't setup your ssh keys yet? [21:15:30] i cant set up ssh keys in putty [21:16:02] hence now i connect to a private v server via password authentification and connect from there via ssh keys to login.tools [21:16:16] https://support.rackspace.com/how-to/logging-in-with-an-ssh-private-key-on-windows/ [21:16:30] Freddy2001: you can definitely use a private key with putty [21:17:36] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2848796 (10scfc) a:03scfc Ouch! It's so "obvious" (probably): `suggestbot` runs a couple of tasks per `crontab` by directly calling `qsub`. These get executed on `tools-webgrid-lighttp... [21:18:55] Freddy2001: besides the link that chasemp gave, here's PuTTY's own docs on using ssh keys -- https://the.earth.li/~sgtatham/putty/0.67/htmldoc/Chapter8.html#pubkey [21:19:38] yeah, I used to be stuck using putty every day and I'm sure it's possible :) Granted years ago! [21:20:16] heh. Me too. Probably ~10 years since I was a regular PuTTY user. [21:21:10] I bet you remember using cygwin and that awkward hate and love feeling :) [21:22:42] yeah loads of cygwin and then later a posix layer for NT 4.5 [21:23:17] unix services for windows? or whatever it was called [21:24:05] you doint need putty any more, git for windows has ssh built in. [21:32:31] chasemp: yeah something like that. I remember being really happy about it but not why ;) [21:37:15] 06Labs: Increase resource quota for dwl - https://phabricator.wikimedia.org/T152456#2848878 (10Giftpflanze) [21:44:34] 06Labs: Increase resource quota for dwl - https://phabricator.wikimedia.org/T152456#2848925 (10Giftpflanze) [22:00:11] 06Labs, 10Tool-Labs, 10crosswatch: Crosswatch sends out large amounts of error mails, crashing tools-mail - https://phabricator.wikimedia.org/T143476#2848993 (10scfc) 05Open>03Invalid I don't think this has happened since August, and @Sitic was last seen on Phabricator and dewp more than a year ago, so t... [22:12:37] 06Labs, 07Wikimedia-Incident: Monitor labs new instance creation - https://phabricator.wikimedia.org/T123590#2849086 (10greg) [22:26:20] (03PS1) 10BryanDavis: jsub: Fix #!... to actually work [labs/toollabs] - 10https://gerrit.wikimedia.org/r/325431 (https://phabricator.wikimedia.org/T147350) [22:30:49] 10Labs-project-Wikistats: allthetropes is not updating on wikistats - https://phabricator.wikimedia.org/T146712#2849146 (10Dzahn) a:03Dzahn [22:35:52] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Enable HTTP based service checks for k8s webservices - https://phabricator.wikimedia.org/T139157#2421136 (10scfc) [22:35:55] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Enable mod_status by default in lighttpd-webservice & make that a http health check - https://phabricator.wikimedia.org/T139158#2849149 (10scfc) 05Open>03declined a:03yuvipanda >>! In T139157#2574495, @yuvipanda wrote: > I'm going to not do this unless someone sp... [22:36:02] 10Labs-project-Wikistats: allthetropes is not updating on wikistats - https://phabricator.wikimedia.org/T146712#2849155 (10Dzahn) When i follow the links in the "good" column: http://allthetropes.miraheze.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&maxlag=5 http://bus.miraheze.org/w/api.php?acti... [22:38:22] 10Labs-project-Wikistats: allthetropes is not updating on wikistats - https://phabricator.wikimedia.org/T146712#2849156 (10labster) You chose the worst day to start working on this as our db server decided to crash today. I'll get back to you when we're back online. [22:39:34] (03CR) 10BryanDavis: [C: 032] jsub: Fix #!... to actually work [labs/toollabs] - 10https://gerrit.wikimedia.org/r/325431 (https://phabricator.wikimedia.org/T147350) (owner: 10BryanDavis) [22:43:44] (03Merged) 10jenkins-bot: jsub: Fix #!... to actually work [labs/toollabs] - 10https://gerrit.wikimedia.org/r/325431 (https://phabricator.wikimedia.org/T147350) (owner: 10BryanDavis) [22:50:14] !log Updated jobutils to 1.17 on tools-bastion-02 (T147350) [22:50:15] Unknown project "Updated" [22:50:15] T147350: Change Python hashbang to `#! /usr/bin/env python -E -s` for user-facing tools - https://phabricator.wikimedia.org/T147350 [22:50:58] !log Updated jobutils to 1.17 on tools-bastion-03 (T147350) [22:50:59] Unknown project "Updated" [22:52:37] !log Updated jobutils to 1.17 on tools-cron-01 (T147350) [22:52:37] Unknown project "Updated" [22:52:46] !log tools Updated jobutils to 1.17 on tools-bastion-02 (T147350) [22:52:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:52:56] !log tools Updated jobutils to 1.17 on tools-bastion-03 (T147350) [22:53:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:53:08] !log tools Updated jobutils to 1.17 on tools-cron-01 (T147350) [22:53:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:53:41] !log tools Updated jobutils to 1.17 on tools-precise-dev (T147350) [22:53:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:55:10] !log tools Updated jobutils to 1.17 on tools-mail (T147350) [22:55:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:09:14] 06Labs, 10Tool-Labs: Change ordering of image flavors in wikitech for tools - https://phabricator.wikimedia.org/T142167#2849233 (10scfc) 05Open>03Resolved https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=create&project=tools®ion=eqiad shows `debian-8.6-jessie` as default. [23:19:22] !log tools Updated toollabs-webservice to 0.31 on tools-bastion-02 (T147350) [23:19:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:19:26] T147350: Change Python hashbang to `#! /usr/bin/env python -E -s` for user-facing tools - https://phabricator.wikimedia.org/T147350 [23:35:44] 06Labs, 10The-Wikipedia-Library: Change URL from twl-test.wmflabs.org to wikipedialibrary.wmflabs.org - https://phabricator.wikimedia.org/T152468#2849325 (10Samwalton9) [23:36:17] 06Labs, 10The-Wikipedia-Library: Change URL from twl-test.wmflabs.org to wikipedialibrary.wmflabs.org - https://phabricator.wikimedia.org/T152468#2849325 (10Samwalton9) @Aklapper I'm not exactly sure who to direct this to, so assistance would be appreciated. Thanks. [23:52:12] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]