[00:59:47] andrewbogott: thanks [02:11:23] Hello all, I need a spot of help getting onto Labs. [02:11:33] I swear every time I want to work on something there it's something new again. [02:11:40] Always my own fault too... [02:12:05] I overwrote the SSH keygen on my development machine, and it's now prompting me for a passphrase. [02:12:43] Is reseting my longin informatio online as simple as repeating a few steps? [03:41:43] PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL 30.00% of data above the critical threshold [0.0] [03:41:57] PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL 30.00% of data above the critical threshold [0.0] [03:43:39] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL 20.00% of data above the critical threshold [0.0] [03:45:17] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL 22.22% of data above the critical threshold [0.0] [03:45:18] PROBLEM - Puppet failure on tools-submit is CRITICAL 22.22% of data above the critical threshold [0.0] [03:45:30] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 22.22% of data above the critical threshold [0.0] [03:46:40] PROBLEM - Puppet failure on tools-shadow is CRITICAL 30.00% of data above the critical threshold [0.0] [03:46:54] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL 40.00% of data above the critical threshold [0.0] [03:47:02] PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL 40.00% of data above the critical threshold [0.0] [03:47:30] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL 44.44% of data above the critical threshold [0.0] [03:47:52] YuviPanda: those are caused by 'Warning: Error 400 on SERVER: No such file or directory - /var/lib/puppet/yaml/node/tools-exec-1214.tools.eqiad.wmflabs.yaml20150801-25191-vtng1.lock' [03:47:57] and the like [03:48:27] ah, disk full on labcontrol1001 [03:48:30] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL 44.44% of data above the critical threshold [0.0] [03:49:22] PROBLEM - Puppet failure on tools-mail is CRITICAL 55.56% of data above the critical threshold [0.0] [03:49:58] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 60.00% of data above the critical threshold [0.0] [03:50:02] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL 30.00% of data above the critical threshold [0.0] [03:50:42] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL 20.00% of data above the critical threshold [0.0] [03:51:00] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 30.00% of data above the critical threshold [0.0] [03:53:15] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 22.22% of data above the critical threshold [0.0] [03:54:31] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL 40.00% of data above the critical threshold [0.0] [03:54:53] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL 20.00% of data above the critical threshold [0.0] [03:55:37] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL 20.00% of data above the critical threshold [0.0] [03:55:37] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 40.00% of data above the critical threshold [0.0] [03:56:44] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL 60.00% of data above the critical threshold [0.0] [03:56:50] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL 50.00% of data above the critical threshold [0.0] [03:57:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL 44.44% of data above the critical threshold [0.0] [04:06:43] RECOVERY - Puppet failure on tools-exec-1214 is OK Less than 1.00% above the threshold [0.0] [04:20:36] RECOVERY - Puppet failure on tools-exec-1406 is OK Less than 1.00% above the threshold [0.0] [04:21:55] RECOVERY - Puppet failure on tools-exec-1201 is OK Less than 1.00% above the threshold [0.0] [04:23:29] RECOVERY - Puppet failure on tools-exec-gift is OK Less than 1.00% above the threshold [0.0] [04:23:37] RECOVERY - Puppet failure on tools-exec-1217 is OK Less than 1.00% above the threshold [0.0] [04:24:19] RECOVERY - Puppet failure on tools-mail is OK Less than 1.00% above the threshold [0.0] [04:24:59] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK Less than 1.00% above the threshold [0.0] [04:25:13] RECOVERY - Puppet failure on tools-submit is OK Less than 1.00% above the threshold [0.0] [04:25:23] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK Less than 1.00% above the threshold [0.0] [04:25:29] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK Less than 1.00% above the threshold [0.0] [04:26:43] RECOVERY - Puppet failure on tools-shadow is OK Less than 1.00% above the threshold [0.0] [04:26:51] RECOVERY - Puppet failure on tools-exec-1208 is OK Less than 1.00% above the threshold [0.0] [04:26:59] RECOVERY - Puppet failure on tools-exec-1212 is OK Less than 1.00% above the threshold [0.0] [04:27:32] RECOVERY - Puppet failure on tools-exec-1216 is OK Less than 1.00% above the threshold [0.0] [04:29:58] RECOVERY - Puppet failure on tools-exec-cyberbot is OK Less than 1.00% above the threshold [0.0] [04:30:46] RECOVERY - Puppet failure on tools-exec-1209 is OK Less than 1.00% above the threshold [0.0] [04:31:02] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK Less than 1.00% above the threshold [0.0] [04:31:48] RECOVERY - Puppet failure on tools-exec-1215 is OK Less than 1.00% above the threshold [0.0] [04:31:48] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK Less than 1.00% above the threshold [0.0] [04:32:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK Less than 1.00% above the threshold [0.0] [04:33:12] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK Less than 1.00% above the threshold [0.0] [04:34:28] RECOVERY - Puppet failure on tools-exec-1207 is OK Less than 1.00% above the threshold [0.0] [04:34:53] RECOVERY - Puppet failure on tools-webproxy-02 is OK Less than 1.00% above the threshold [0.0] [04:35:37] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0] [08:05:43] RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [3600.0] [08:06:39] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [0.0] [08:57:40] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL 33.33% of data above the critical threshold [0.0] [09:37:40] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [0.0] [09:42:43] How does Labs handle redirects from one instance to another? [09:43:38] Nemo_bis: redirects in what sense? http redirects are handled fully client side [09:43:50] do you mean the proxy system? [09:44:46] what do you mean by client [09:45:22] I need the labs equivalent of operations/puppet/modules/mediawiki/files/apache/sites/redirects [09:52:37] client = the browser [09:53:02] do you mean the apache configuration for beta? [10:07:32] PROBLEM - Puppet failure on tools-static-01 is CRITICAL 100.00% of data above the critical threshold [0.0] [10:10:31] 10Wikibugs, 6Phabricator, 5Patch-For-Review: Set up dumping Phabricator's project taxonomy to a wiki - https://phabricator.wikimedia.org/T85096#1500526 (10Nemo_bis) [11:40:18] valhallasw`cloud: no, for all of labs [11:40:45] ??? [11:40:59] is there a frontend which can redirect requests before they reach individual instances? otherwise I have to hunt the configuration of individual domains, which is hard [11:41:15] I guess I'd better just mention the goal at hand: https://phabricator.wikimedia.org/T104545 [11:41:16] not all requests pass through the same systems [11:41:24] I need to redirect one instance URL to another instance [11:42:19] add a redirect entry on the server serving metrics.wmflabs.org? [11:42:50] the labs proxy just redirects all HTTP requests to .wmflabs.org to a certain instance [11:43:09] that configuration is done on wikitech, I'm not sure where it's stored (probably LDAP) [13:28:39] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:03:40] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [0.0] [14:13:01] YuviPanda: still aboard? [14:34:16] 10Tool-Labs-tools-Erwin's-tools: relatedchanges.php MySQL errors - https://phabricator.wikimedia.org/T107618#1500635 (10Nemo_bis) 5Open>3Resolved a:3Nemo_bis Ok. The origin of the bug is very silly: TSDatabase::getDatabase() and TSDatabase::getCluster() had a line: `$sql = "SELECT dbname FROM meta_p.wiki W... [14:40:34] 10Tool-Labs-tools-Erwin's-tools: Restore categorycount and catanalyzer - https://phabricator.wikimedia.org/T107656#1500640 (10Nemo_bis) 3NEW [14:42:57] 10Tool-Labs-tools-Erwin's-tools: Restore categorycount and catanalyzer - https://phabricator.wikimedia.org/T107656#1500649 (10Nemo_bis) 5Open>3Resolved a:3Nemo_bis [14:44:55] 6Labs, 10Tool-Labs: /srv on tools-static-01 running out of inodes - https://phabricator.wikimedia.org/T107657#1500651 (10Andrew) 3NEW [15:18:05] (03PS1) 10Gerrit Patch Uploader: Explicitly include error files [labs/toollabs] - 10https://gerrit.wikimedia.org/r/228485 (https://phabricator.wikimedia.org/T85738) [15:18:07] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/228485 (https://phabricator.wikimedia.org/T85738) (owner: 10Gerrit Patch Uploader) [15:19:22] YuviPanda / Coren / andrewbogott ^. Also, if scfc and myself could be added as +2'ers for labs/toollabs, that'd be awesome. [16:13:24] 10Tool-Labs-tools-Erwin's-tools: Restore delete.php - https://phabricator.wikimedia.org/T107663#1500714 (10Nemo_bis) 3NEW a:3Nemo_bis [16:13:35] 10Tool-Labs-tools-Erwin's-tools: Restore delete.php - https://phabricator.wikimedia.org/T107663#1500722 (10Nemo_bis) 5Open>3Resolved [16:16:52] valhallasw`cloud: agree ya. I'll add you guys once I get to a laptop [16:17:19] andrewbogott: yes! I had slept [16:23:30] valhallasw`cloud: I gave you and Tim +2 [16:23:33] check? [16:25:23] 10Tool-Labs-tools-Erwin's-tools: Restore delete.php - https://phabricator.wikimedia.org/T107663#1500736 (10Nemo_bis) [16:33:55] Playing a game now [16:46:26] (03CR) 10Merlijn van Deen: [C: 032] Explicitly include error files [labs/toollabs] - 10https://gerrit.wikimedia.org/r/228485 (https://phabricator.wikimedia.org/T85738) (owner: 10Gerrit Patch Uploader) [16:46:33] YuviPanda: ^ \o/ [16:46:38] except I probably need to V+2 as well [16:46:45] no, jenkins is thre [16:51:52] YuviPanda: except it's not being merged :S [17:09:02] valhallasw`cloud: maybe I need to give you submit right specifically [17:09:05] I'll check in a bi [17:09:07] Bit [17:12:27] 6Labs, 10Tool-Labs: /srv on tools-static-01 running out of inodes - https://phabricator.wikimedia.org/T107657#1500753 (10scfc) The underlying problem is in `modules/toollabs/manifests/static.pp`: ``` labs_lvm::volume { 'cdnjs-disk': mountat => '/srv', size => '100%FREE' } ``` This... [17:15:13] 6Labs, 10Tool-Labs: create diamond reporter & shinken alert for /var/log/account/pacct and/or pacct.1 size - https://phabricator.wikimedia.org/T107617#1500755 (10scfc) I think this is too much of a fringe case to guard against. If it happens, it already triggers the file system alerts, and then it is (usually... [17:28:58] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL 30.00% of data above the critical threshold [0.0] [17:31:05] 6Labs, 10Tool-Labs: Determine and deploy proper h_vmem resources for execution nodes - https://phabricator.wikimedia.org/T107665#1500758 (10scfc) 3NEW [17:45:57] (03CR) 10Tim Landscheidt: "@valhallasw: Could you please upload a PS2 with the author set to yourself? That's much better to read in the Git history." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/228485 (https://phabricator.wikimedia.org/T85738) (owner: 10Gerrit Patch Uploader) [18:04:34] (03CR) 10Merlijn van Deen: "as far as I can see, the author is set to myself? The committer is set to gerrit patch uploader because Gerrit requires this." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/228485 (https://phabricator.wikimedia.org/T85738) (owner: 10Gerrit Patch Uploader) [18:08:59] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK Less than 1.00% above the threshold [0.0] [18:09:32] !log tools depooling/rebooting tools-webgrid-lighttpd-1407 because it’s unable to fork [18:09:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, dummy [18:09:41] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL 37.50% of data above the critical threshold [0.0] [18:14:07] tools instances being very needy today [18:24:42] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [0.0] [18:26:01] 6Labs, 6operations, 3Labs-Sprint-107, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1500789 (10Andrew) Reboot of labvirt1009 is now scheduled and announced for Wednesday. [18:27:05] (03CR) 10Tim Landscheidt: "I'm sorry, you're right. I only looked at the top of the Gerrit page and the bot's first comment." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/228485 (https://phabricator.wikimedia.org/T85738) (owner: 10Gerrit Patch Uploader) [18:35:18] andrewbogott: note that not all jobs can be restarted -- just continuous ones are marked as restartable [18:35:49] (but there's no exec nodes in the list you posted) [19:12:02] andrewbogott: i understand you migrated encoding01 [19:12:17] matanya: I did — was it damaged? [19:12:43] valhallasw`cloud: is that re: my email about rebooting labvirt1009? [19:12:46] andrewbogott: no, seem ok [19:13:05] matanya: ok. Sorry about the lack of warning… that server was in danger of dying so I was Bold [19:13:26] andrewbogott: no worries, it is rare anything is running there on the weekend [19:13:50] i work on it mostly sunday-friday morning [19:13:59] (my time) [19:37:44] YuviPanda: My query results aren't showing http://quarry.wmflabs.org/query/894 [21:18:23] 6Labs, 10Tool-Labs: Citations button causes Bad Request error - https://phabricator.wikimedia.org/T107649#1500904 (10Multichill) [21:19:34] andrewbogott: yes [21:21:21] hello labs [21:21:59] 6Labs, 10Tool-Labs: Citations button causes Bad Request error - https://phabricator.wikimedia.org/T107649#1500907 (10Multichill) Looks like https://en.wikipedia.org/wiki/MediaWiki:Gadget-citations.js [21:24:34] 6Labs, 10Tool-Labs: Citations button causes Bad Request error - https://phabricator.wikimedia.org/T107649#1500911 (10Multichill) So https://en.wikipedia.org/wiki/MediaWiki:Gadget-citations seems to be the info place. Users in charge of this tool are Smith609, Mattsenate, and Maximilianklein [21:27:05] 6Labs, 10Tool-Labs: Citations button causes Bad Request error - https://phabricator.wikimedia.org/T107649#1500913 (10valhallasw) Based on error.log, this is an error in the actual code, so the tool developers (mattsenate, maximilianklein and smith609) will have to take a look at it. ``` 2015-08-01 21:19:04: (... [21:30:21] valhallasw`cloud: I pinged the maintainers at https://en.wikipedia.org/wiki/MediaWiki_talk:Gadget-citations.js#Bug [21:30:49] multichill: thanks [21:34:42] valhallasw`cloud: gave you two explicit submit rights [21:35:08] andrewbogott: you have to approve the labs-l emails I think [21:35:14] * YuviPanda doesn't have the password [21:35:22] YuviPanda: thanks [21:35:37] valhallasw`cloud: check? [21:35:58] scfc submitted it, so I guess it works ;-) [21:36:04] (no other changes to submit) [21:36:17] but I'm mainly confused jenkins didn't submit [21:36:32] (03PS1) 10Sitic: Fix watchlist padding [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228567 [21:36:34] (03PS1) 10Sitic: Add more de transations [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228568 [21:36:36] (03PS1) 10Sitic: Fix translation of lists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228569 (owner: 10Sitic) [21:37:36] (03CR) 10Sitic: [C: 032 V: 032] Add translation file versioning [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228570 (owner: 10Sitic) [21:38:11] valhallasw`cloud: I don't think it's set up for that [22:30:40] 10Tool-Labs-tools-nlwikibots: e85bot / Tvpmelder does not work in all months - https://phabricator.wikimedia.org/T68347#1500951 (10Akoopal) 5Open>3Resolved a:3Akoopal Fixed by setting the locale in the script: locale.setlocale(locale.LC_ALL, 'nl_NL.UTF-8') [22:36:01] 6Labs, 10Tool-Labs: Citations button causes Bad Request error - https://phabricator.wikimedia.org/T107649#1500954 (10Krenair) >>! In T107649#1500892, @Greenrd wrote: > Developers can do anything so you can just log in as me and see what I see, surely. Not "just", no. If that was easy and non-hacky it'd be som... [22:42:31] is there a log of become? [22:45:51] you want a log of people logging in as a certain tool? [22:50:39] or f [22:50:52] * or of wannabes [22:54:48] I wanted to see who logged in on a certain tool to see who fixed what I was going to look at [22:54:58] thanks btw Nemo_bis :-) [22:55:11] valhallas found it out for me [23:01:55] hah :) [23:02:26] we should really set up a repo for that tool [23:40:00] sitic crosswatch looks pretty great [23:46:14] cross post: so I am unsure if this is right place to ask this but, whats the procedure for closed wikis? [23:49:58] ToAruShiroiNeko: What procedure? [23:51:19] YuviPanda: :-) [23:52:04] what is done if a decision is made to close a wiki? [23:52:23] do you run an sql command, dance with a rubber chicken? what? :) [23:52:41] I want to understand the technical actions taken [23:53:43] You pull out a gratuitously large weapon and hack/blast the server responsible to pieces [23:53:54] Then repair collateral damage [23:54:42] Probably have to move the SQL database to the incubator or whatever and setup web server to not go to the old wiki [23:55:54] oh I am not worried about the data [23:56:04] I just care about how to lock and unlock a wiki [23:56:11] I find the existing procedure a bit problematic [23:56:57] Before I propose a change to that I want to fully understand what I am dealing with