[00:58:29] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:23:29] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [02:19:30] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:04:29] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [03:55:30] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:25:31] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [04:36:30] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:57:44] why I get Connection closed by? [05:22:25] Mjbmr: From SSH? [05:22:30] Logging onto where? [05:24:04] Reedy: I login to a bastion instance or tools-login and I try to login to other instance from there, connections getting closed after authorization pass, do even I allow login to other instances? where I'm supposed to run bot scripts? [05:24:40] which instance(s)? [05:25:28] any other instance listed in [[Nova_Resource:Tools]] with non-public IP. [05:26:29] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [05:26:46] I don't think you need to login to any other machine bar tools-login [05:26:52] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs [05:26:56] https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver [05:27:26] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Submitting.2C_managing_and_scheduling_jobs_on_the_grid [05:28:44] I've searching for what I'm suppose to do, but I though I can login to another server with more resources? is it ok to run a bot script on main tools-login? [05:30:01] You don't run jobs directly on tools-login [05:30:14] Certainly not long running intensive jobs at least [05:30:47] Mjbmr: you should use https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid [05:30:55] for running them [05:31:06] also, is it possible login using tools usernames directly? [05:31:16] no [05:31:50] why not, how I'm supposed to edit files with filezilla? [05:32:18] you should be able to browse the directories as long as the permissions are correct [05:32:26] same as it was on toolserver [05:41:31] Thank you [06:17:32] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:26:34] hoi what is going on [06:26:39] https://tools.wmflabs.org/toolscript/index.html?pastebin=x9TXBfZi does not serve [06:26:54] oAuth is up shit creek [06:27:04] what is going on [06:37:26] it gets me a 504 Gateway Time-out [06:39:18] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:42:32] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [06:44:38] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:04:13] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:14:42] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:35] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:43:30] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [07:54:35] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:24:37] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [08:35:33] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:05:30] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [09:16:32] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:48:10] !log tools restarted toold-webgrid-03 [09:48:17] Logged the message, Master [10:06:30] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:29] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:19:20] RECOVERY - Free space - all mounts on tools-exec-14 is OK: OK: All targets OK [10:27:30] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [10:31:28] RECOVERY - Host tools-webgrid-generic-01 is UP: PING OK - Packet loss = 0%, RTA = 2.08 ms [10:32:20] PROBLEM - Puppet staleness on tools-webgrid-generic-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [10:33:31] !log manually started tools-webgrid-generic-01 [10:33:32] manually is not a valid project. [10:34:28] !log tools manually started tools-webgrid-generic-01 [10:34:31] Logged the message, Master [10:42:19] RECOVERY - Puppet staleness on tools-webgrid-generic-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [13:39:17] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<22.22%) [13:49:16] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [13:55:16] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<22.22%) [14:15:16] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [14:56:19] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%) [15:16:18] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [15:22:15] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<22.22%) [15:47:17] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [18:25:12] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:07:00] 3Analytics-Engineering, Tool-Labs: Copy paid MaxMind geolocation dbs to tool labs - https://phabricator.wikimedia.org/T87151#985744 (10yuvipanda) [20:11:35] 3Analytics-Engineering, Tool-Labs: Copy paid MaxMind geolocation dbs to tool labs - https://phabricator.wikimedia.org/T87151#985763 (10Ironholds) That's the "if we're able to do it". I'll CC Luis. [20:11:43] 3Analytics-Engineering, Tool-Labs: Copy paid MaxMind geolocation dbs to tool labs - https://phabricator.wikimedia.org/T87151#985765 (10Ironholds) [20:15:23] 3Analytics-Engineering, Tool-Labs: Copy paid MaxMind geolocation dbs to tool labs - https://phabricator.wikimedia.org/T87151#985768 (10yuvipanda) Also, we could copy the *free* one there, if the paid one isn't something we can legally copy :) Remember putting it on labs is pretty much making it available to ever... [20:24:26] 3Analytics-Engineering, Tool-Labs: Copy paid MaxMind geolocation dbs to tool labs - https://phabricator.wikimedia.org/T87151#985783 (10faidon) For what it's worth, I think the license for the databases is https://www.maxmind.com/en/license_agree and while IANAL (L being Luis here ;)), there are several clauses t... [20:47:03] 3Labs-Team: New disk partition scheme for labs instances - https://phabricator.wikimedia.org/T87003#985808 (10yuvipanda) Would be nice to get this done soonish. I've to re-image tools-redis sooon [20:51:22] YuviPanda, have you tried !log yet? :> [20:51:29] haha [20:51:34] !log tools because valhallasw is nice [20:51:38] Logged the message, Master [20:51:43] ...hmpf. [20:51:43] well? [20:51:45] did it crash? [20:52:05] I started a version with the fancy url-to-log-page code [20:52:35] !log tools.wikibugs is this really broken? :( [20:52:37] Logged the message, Master [20:52:41] clearly. [20:54:58] RECOVERY - Puppet staleness on tools-redis is OK: OK: Less than 1.00% above the threshold [3600.0] [20:58:27] YuviPanda: hm, not sure what happened. it was restarted really quickly after an ERROR: Died in main event loop [21:00:05] 3Quarry: The last run time of all queries reports 5 months ago - https://phabricator.wikimedia.org/T87200#985819 (10Halfak) 3NEW