[01:25:15] 6Labs, 10Tool-Labs, 7Database, 3labs-sprint-117: tools.citationhunt can't access databases - https://phabricator.wikimedia.org/T109972#1707654 (10yuvipanda) Ok, so this has happened now and I've recreated it with a user account / and new replica.my.cnf. @Surlycyborg can you confirm everything is ok? [01:57:02] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:37:00] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [02:57:18] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:32:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [05:01:15] off the wall question that I can’t find in the docs. Is there any way to set up sharing a database between two tools? [05:32:18] 6Labs, 10Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#1707857 (10Dzahn) how about a different kind of icinga.wmflabs.org ? one that is not setup manually and does not monitor labs instances, but that is there to test changes on pr... [05:58:13] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:26:24] 6Labs, 10Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#1707892 (10yuvipanda) Volunteers welcome :) I'll add whoever asks to the icinga project if they want to spend time untangling our icinga puppet code :) [06:27:26] 6Labs, 10Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#1707894 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Volunteers welcome :) I'll add whoever asks to the icinga project if they want to spend time untangling our icinga pupp... [06:33:19] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [08:59:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [10:04:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [10:55:21] How do I set session.save_path for a tool? [11:02:58] Nemo_bis: either by overriding the php fcgi configuration (nontrivial), or by calling https://secure.php.net/manual/en/function.session-save-path.php before you call session_start() [13:25:12] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:05:15] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:27] 6Labs, 10Labs-Infrastructure: Labs: both Icinga and Ganglia not accessible: 502 Bad Gateway - https://phabricator.wikimedia.org/T85318#1708846 (10Dzahn) @yuvipanda yes, we know the icinga puppet code is tangled. the point was that people actually do that (example https://gerrit.wikimedia.org/r/#/q/owner:jzereb... [14:51:10] 6Labs, 10Tool-Labs: proxylistener errors on tools-proxy-01 - https://phabricator.wikimedia.org/T114223#1708894 (10Aklapper) #Labs team (@yuvipanda, @coren, @Andrew): Do you know who is supposed to take a look at this? This has been "[[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task... [15:06:48] can anyone help me get the delete right on wikitech? [15:07:13] We (analytics) wanted to do some documentation cleanup and it's still confusing if we just slap "archived" on top of all the out of date pages [15:09:42] Coren: is that something you do (give people delete rights ^) [15:29:04] 6Labs, 10Labs-Infrastructure, 6operations, 3labs-sprint-117: add logrotate for designate logs (holmium disk space) - https://phabricator.wikimedia.org/T114544#1709038 (10Andrew) 5Open>3Resolved Looks like it's working. [15:29:50] milimetric: looks like you want "contentadmin" [15:30:03] What username? [15:31:23] Reedy: https://wikitech.wikimedia.org/wiki/User:Milimetric I'd assume [15:31:49] Reedy: yes, milimetric [15:31:54] thx JohnFLewis [15:32:11] Never quite know/remember when people use other names etc [15:32:25] Done [15:32:26] Reedy: we also need this on mediawiki, user Milimetric (WMF), how would I go about that? [15:32:35] mediawiki.org? [15:32:37] yes [15:33:23] Done there too [15:33:36] thx very much Reedy [15:58:54] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1709164 (10mark) If we have hardware that is out of warranty and (therefore) won't be used for new production stuff, then it could be considered "free"... [16:19:55] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1709234 (10Andrew) [16:24:42] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1709246 (10RobH) https://rt.wikimedia.org/Ticket/Display.html?id=9677 is the rt ticket to track the quoting of a 1u misc system for pricing consideratio... [16:27:13] milimetric: I can give you admin-like. [16:27:51] milimetric: ... you already have it. [16:28:09] Ah, Reedy ninja'd me. [16:41:00] Coren or yuvipanda, do you know what’s up with this? https://phabricator.wikimedia.org/T114223 [16:42:00] andrewbogott: I would have thought Yuvi was on it, but I'll take a look - it's a good opportunity to figure out the setup. [16:42:07] thanks [16:54:21] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1709361 (10Thgoiter) [16:56:18] (03CR) 10Legoktm: [C: 032] Also exclude TCB-Team- from #wikimedia-fundraising [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/241652 (owner: 10Addshore) [16:56:27] 6Labs, 10Tool-Labs: proxylistener errors on tools-proxy-01 - https://phabricator.wikimedia.org/T114223#1709373 (10coren) a:3coren [16:56:41] (03Merged) 10jenkins-bot: Also exclude TCB-Team- from #wikimedia-fundraising [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/241652 (owner: 10Addshore) [16:57:14] !log tools.wikibugs Updated channels.yaml to: c78efa6e621b316c26fca060661f59558e8bafa5 Merge "Also exclude TCB-Team- from #wikimedia-fundraising" [16:57:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [17:03:47] 6Labs, 10Tool-Labs: proxylistener errors on tools-proxy-01 - https://phabricator.wikimedia.org/T114223#1709415 (10coren) 5Open>3Resolved This was fixed around Oct 1 06:51:32 UTC, apparently by @yuvipanda as he was working on the kubernetes proxy during that period. [17:36:30] andrewbogott: yuvipanda: I'm on the labstore load [17:36:30] thanks [17:36:30] One of the lighttpds. [17:38:34] catscan2 [17:42:31] hammer wielded. [17:43:56] 6Labs, 10Tool-Labs: proxylistener errors on tools-proxy-01 - https://phabricator.wikimedia.org/T114223#1709584 (10yuvipanda) @Joe was probably responsible :P But not sure why the proxylistener would've errored out, we didn't touch it. [17:44:44] 6Labs, 10Tool-Labs: proxylistener errors on tools-proxy-01 - https://phabricator.wikimedia.org/T114223#1709591 (10coren) I think the root cause was puppet not running; at some point it was restarted which seems to have fixed the issue. [17:45:25] 6Labs, 10Tool-Labs: proxylistener errors on tools-proxy-01 - https://phabricator.wikimedia.org/T114223#1709593 (10yuvipanda) ok! [18:22:50] Coren: did I perchance break your session on wikietch? Can you still view instance lists? [18:29:34] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review, 3labs-sprint-117: Give 'novaobserver' keystone account rights to read everything, everywhere, write or change nothing - https://phabricator.wikimedia.org/T104588#1709796 (10Andrew) I just upgraded us to keystone v3 api, which should allow us to use domains. [18:39:03] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1709819 (10Andrew) [19:14:39] andrewbogott: You have, and I can't. [19:39:14] Coren: ok, I’ll do a reset [19:56:13] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:14:48] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Scaling: Support dedicating a specific virt node to a specific nova project - https://phabricator.wikimedia.org/T84989#1710153 (10hashar) 5Open>3stalled p:5Low>3Lowest [20:15:43] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Scaling: Support dedicating a specific virt node to a specific nova project - https://phabricator.wikimedia.org/T84989#936078 (10hashar) If CI starts to cause troubles to other projects on labs, we will have to look at dedicated hardware for it. There is... [20:31:16] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [20:38:33] yuvipanda: Take a peek at https://gerrit.wikimedia.org/r/#/c/239377/ when you have a minute? [20:38:49] sure [20:38:50] looking now [20:39:09] I think that addresses both your notes. [20:39:21] Coren: yup! [20:39:23] thanks [20:39:25] and sorry about having it drag [20:39:31] Heh. No worries. [21:01:04] 6Labs, 10wikitech.wikimedia.org, 3Labs-Sprint-105: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1710367 (10ori) @Andrew, you don't actually reduce complexity by not using nutcracker on wikitech; you simply shift it around and make it somebody else's problem. Right now, it is mi... [21:14:20] 6Labs, 10Tool-Labs, 7Database, 3labs-sprint-117: tools.citationhunt can't access databases - https://phabricator.wikimedia.org/T109972#1710420 (10yuvipanda) Yw! And sorry it took this long... [21:33:58] a labs admin on? [21:35:45] hi White_Master [21:36:01] just ask! :) [21:36:18] yuvipanda, in pm please :) [22:04:30] Hi [22:05:18] 6Labs, 10wikitech.wikimedia.org, 3Labs-Sprint-105, 5Patch-For-Review: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1710694 (10ori) 5Open>3declined a:3ori [22:44:36] yuvipanda: if I remove role::labs::vagrant from NovaPuppetGroup will hosts that have it applied continue to have it applied? (Meaning will I break the 51 hosts that are still using it?) [22:44:56] bd808: I think so but let me verify [22:46:06] bd808: yup. http://tools.wmflabs.org/watroles/role/role::mediawiki-install::labs is all installs with the old mediawiki-install and they're still there [22:46:36] sweet. I'm going to update some docs and then hide the old role from everyone then [22:46:45] bd808: \o/ [22:46:59] death to labs-vagarnt; long live mediawiki-vagrant [22:47:03] +1 [22:47:10] it was a nice hack for its time [22:47:21] I totally was [22:54:27] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1710998 (10RobH) https://rt.wikimedia.org/Ticket/Display.html?id=9677 now has updated pricing info for a new single cpu misc system. DO NOT PUT PRICIN... [22:57:16] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:27:58] 10Tool-Labs-tools-Global-user-contributions, 6Collaboration-Team-Backlog, 10Flow, 10xTools-on-Labs: Add Flow contributions to GUC and Xtools - https://phabricator.wikimedia.org/T114777#1711094 (10Catrope) p:5Triage>3High [23:28:25] 10Tool-Labs-tools-Global-user-contributions, 6Collaboration-Team-Backlog, 10Flow, 10xTools-on-Labs: Add Flow contributions to GlobalUserContribution and Xtools - https://phabricator.wikimedia.org/T114777#1711097 (10Mattflaschen) [23:28:31] 10Tool-Labs-tools-Global-user-contributions, 6Collaboration-Team-Backlog, 10Flow, 10xTools-on-Labs: Add Flow contributions to GlobalUserContributions and Xtools - https://phabricator.wikimedia.org/T114777#1705954 (10Mattflaschen) [23:32:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0]