[03:48:48] Thanks, shinken-wm. [03:48:53] How do I add an instance? [03:48:58] I got to this page: https://wikitech.wikimedia.org/wiki/Nova_Resource:Butterfly [03:49:07] I don't see any links to add or delete an instance. [04:36:19] Debra: see https://wikitech.wikimedia.org/wiki/Special:NovaInstance to add/delete or use the new interface at https://horizon.wikimedia.org/ [04:38:21] bd808: Yeah, I started at https://wikitech.wikimedia.org/wiki/Special:NovaInstance and filtered to two projects. [04:38:30] I clicked on "butterfly" and went to https://wikitech.wikimedia.org/wiki/Nova_Resource:Butterfly [04:38:50] Is the wiki interface for managing instances dead? [04:40:14] Oh. I bet you are getting the bug where your session to wikitech is alive but the session with the open stack backend has expired. Try logging out and logging back in. You should see the existing instances and the create link on https://wikitech.wikimedia.org/wiki/Special:NovaInstance after picking a project. [04:41:57] I actually had to do the logout/login dance myself just now when checking to see if things were really broken [04:42:00] http://imgur.com/a/Czwwd is what I see currently. [04:42:09] I'll try logging out and logging back in now. [04:43:01] *nod* that screen shot matches what happens when the sessions aren't in sync [04:43:03] Oh, that seemed to work. Nice. [04:43:09] Cool, thanks! [04:48:10] no problem. We've looked for a fix for that several times, but kind of gave up. horizon does not have that problem (but also does not have long lived sessions) [07:54:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Salahyahya was created, changed by Salahyahya link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Salahyahya edit summary: Created page with "{{Tools Access Request |Justification=I want to build ATM's and Branches in Palestine but using the English labels, we used the old server but its not working now https://www...." [08:07:54] (03CR) 10Jean-Frédéric: "> Looks good at a first glance. Will look into it deeper later." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313766 (https://phabricator.wikimedia.org/T145333) (owner: 10Jean-Frédéric) [08:10:29] (03CR) 10Jean-Frédéric: [C: 032] Add Nigeria in English to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) (owner: 10Lokal Profil) [08:14:41] (03CR) 10Jean-Frédéric: Only output one primkey warning per page (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [08:52:13] (03CR) 10Lokal Profil: [C: 032] "> Shall we merge as is (also not to block eg Nigeria) and later" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313766 (https://phabricator.wikimedia.org/T145333) (owner: 10Jean-Frédéric) [08:53:03] (03Merged) 10jenkins-bot: Break down monuments_config in individual JSON files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313766 (https://phabricator.wikimedia.org/T145333) (owner: 10Jean-Frédéric) [08:53:05] (03Merged) 10jenkins-bot: Add Nigeria in English to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314860 (https://phabricator.wikimedia.org/T143573) (owner: 10Lokal Profil) [08:53:07] (03PS8) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [08:53:19] (03CR) 10Lokal Profil: Only output one primkey warning per page (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [08:57:10] (03CR) 10Jean-Frédéric: [C: 032] Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [08:58:47] (03Merged) 10jenkins-bot: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [09:15:42] !log tools.heritage Deployed master from Git: dce7434, 8bfba2a (T145333), faa356e, f87cb70 (T145574), 64ea088 (T143573), 8187df6 (T132641), 26171fe (T138633) [09:15:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [09:15:48] T132641: Standardise SQL formatting and lint .sql files at CI - https://phabricator.wikimedia.org/T132641 [09:15:48] T138633: improve logging - https://phabricator.wikimedia.org/T138633 [09:15:48] T143573: Add Nigeria to database - https://phabricator.wikimedia.org/T143573 [09:15:49] T145333: Break down monuments_config in individual JSON files - https://phabricator.wikimedia.org/T145333 [09:15:49] T145574: update TestGetCommonsCatTemplates to mock _load_wikipedia_commonscat_templates() - https://phabricator.wikimedia.org/T145574 [10:40:28] (03PS1) 10Lokal Profil: Require that orderby parameters are included in requested rows [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315488 (https://phabricator.wikimedia.org/T132644) [11:46:56] 06Labs, 10wikitech.wikimedia.org, 07Technical-Debt: Cleanup ContentHandler deprecated calls in Wikitech specific extensions - https://phabricator.wikimedia.org/T147924#2708611 (10Reedy) [14:49:51] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:24:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:16] 10PAWS, 10Pywikibot-core: How to change the default uploading time in PAWS - https://phabricator.wikimedia.org/T147832#2709380 (10yuvipanda) [17:09:36] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:44:39] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:46:51] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2709618 (10RobH) [18:16:18] PROBLEM - Host tools-exec-cyberbot is DOWN: CRITICAL - Host Unreachable (10.68.16.39) [18:35:57] 06Labs, 10Tool-Labs, 15User-bd808: meetbot is hoarding >1T of logs on the 8T Tools NFS share - https://phabricator.wikimedia.org/T147950#2709949 (10chasemp) [18:36:49] Is there a labs admin around? I've got an instance which has completely stopped responding, all network traffic is just timing out to it. Grafana graphs just abruptly stop with nothing obvious before it, and when I did manage to get into it an hour or so ago, it was showing a load avg of ~100. I hit pause (was aiming for soft reboot...) on horizon, and then soft reboot, but it seems to be stuck "pausing" now. [18:37:06] (accounts-db2 is the instance in question) [18:40:12] stwalkerster: ok I'll look to forceably reboot give me a sec [18:41:19] chasemp: it's running a mysql server, is there any concern with regards to corruption with a force reboot? [18:41:51] I mean, possibly yes depending on what it's doing but if it's nonresponsive [18:41:54] there isn't much else we can do [18:42:43] ok, fair. I guess we just see if it comes back up and deal with it from there. :) [18:43:13] so I see it in some kind of pausing-ish state which has precluded a sane reboot as it's already transitioning [18:43:14] hm [18:45:41] stwalkerster: it should come back around in a minute, I was just on the way out so I'll check back in later [18:45:57] OK, cool, I'll try and ssh into it in a couple then, thanks :) [18:59:46] 06Labs, 10MediaWiki-Vagrant: Git Pull failed for SecurePoll extension using MediaWiki-vagrant - https://phabricator.wikimedia.org/T147958#2710167 (10Huji) [19:06:45] 06Labs, 10MediaWiki-Vagrant: Git Pull failed for SecurePoll extension using MediaWiki-vagrant - https://phabricator.wikimedia.org/T147958#2710155 (10bd808) Is your SecurePoll clone dirty (local edits or on a branch other than master)? The `labs-vagrant` software has been deprecated and unmaintained for quite s... [19:09:18] 06Labs, 10Tool-Labs, 15User-bd808: meetbot is hoarding >1T of logs on the 8T Tools NFS share - https://phabricator.wikimedia.org/T147950#2710177 (10bd808) I cleaned up a lot of crufty looking files: ``` $ webservice --backend=kubernetes stop $ rm access.log $ webservice --backend=kubernetes start $ jstop mee... [19:40:56] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-meetbot, 15User-bd808: meetbot is hoarding >1T of logs on the 8T Tools NFS share - https://phabricator.wikimedia.org/T147950#2710356 (10hashar) [19:45:29] 10Tool-Labs-tools-Pageviews: Pageviews Analysis 2.0 - https://phabricator.wikimedia.org/T147966#2710385 (10MusikAnimal) [20:03:40] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Iislucas was created, changed by Iislucas link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Iislucas edit summary: Created page with "{{Tools Access Request |Justification=Working on the wikidetox project. |Completed=false |User Name=Iislucas }}" [20:34:57] chasemp: any chance you have a quick way to see if an account has log files >100M ? [20:35:43] Betacommand: yes probably as a one-off, what tool? [20:36:17] chasemp: Betacommand-dev [20:39:15] Betacommand: 394M ./project/betacommand-dev/sge_logs/reports_every_30_min.out. I'm unsure if this is ephemeral data or not, that dir in general has some possible suspects [20:39:16] chasemp: thanks, did some house keeping and want to ensure that there isnt unintended collateral from the planned cleanup. [20:39:23] ok thanks :) [20:40:05] chasemp: directory should be empty or near so [20:40:21] great that helps a lot [20:45:41] chasemp: I have my own script for cleanups :P [23:17:33] 06Labs, 10wikitech.wikimedia.org: labstestwiki table doesn't exists, fails Oauth - https://phabricator.wikimedia.org/T147989#2711264 (10Reedy) [23:24:14] 06Labs, 10wikitech.wikimedia.org: labstestwiki table doesn't exists, fails Oauth - https://phabricator.wikimedia.org/T147989#2711368 (10Reedy) 05Open>03Resolved a:03Reedy Both created on labtestwiki