[00:21:15] SMalyshev: I had to reboot wdq-beta, but others should be all good now [00:21:40] yuvipanda: ok, is it done? [00:22:02] SMalyshev: yup [00:22:13] SMalyshev: I moved your homedir to /data/scratch, and can move others as needed [00:22:53] ok, thanks [00:23:10] SMalyshev: np! let me know if anything else was broken [00:23:45] 6Labs, 6Discovery, 10Wikidata, 10Wikidata-Query-Service, 13Patch-For-Review: Move wikidata-query project from /home NFS to scratch - https://phabricator.wikimedia.org/T128815#2097737 (10yuvipanda) 5Open>3Resolved I've moved Stas' homedir to /data/scratch, and have removed /home from all instances. Le... [00:25:18] yuvipanda: seems to be working OK so far [00:25:23] SMalyshev: \o/ cool [00:25:30] SMalyshev: remember /data/scratch is viewable by all tho [00:26:07] yuvipanda: yes, np, there's nothing secret there AFAIK [00:26:14] cool :D [00:41:38] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2097789 (10DannyH) [01:12:51] 6Labs, 10DBA: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2097945 (10Andrew) yep! [02:26:19] !log Updating HHVM on deployment-mediawiki02 [02:26:20] Updating is not a valid project. [02:26:51] !log deployment-prep Updating HHVM on deployment-mediawiki02 [02:26:52] Please !log in #wikimedia-releng for beta cluster SAL [02:26:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [03:19:25] 10PAWS: Allow restarting PAWS hub without taking down all the instances - https://phabricator.wikimedia.org/T128508#2098082 (10yuvipanda) 5Open>3Resolved a:3yuvipanda This is done now! DB is on a mysql setup that's in tools db, and the cookie is stored in the kubernetes config. [03:19:54] 10PAWS, 6Revision-Scoring-As-A-Service: Install revscoring inside PAWS - https://phabricator.wikimedia.org/T120317#2098085 (10yuvipanda) Libraries are installed now, but the models aren't. [03:20:38] 10PAWS: PAWS 404 for users with special characters in their names - https://phabricator.wikimedia.org/T120066#2098086 (10yuvipanda) I spent some more time on this, attempting to reproduce this with a dummy authenticator and spawner. And could not. Booooo! I'll have to spend some more time on this tomorrow [03:20:50] 10PAWS: PAWS 404 for users with special characters in their names - https://phabricator.wikimedia.org/T120066#2098087 (10yuvipanda) p:5Triage>3High a:3yuvipanda [03:21:03] 10PAWS: PAWS 404 for users with special characters in their names - https://phabricator.wikimedia.org/T120066#1844005 (10yuvipanda) Going to make a concerted effort tomorrow to figure this one out. [03:21:38] 10PAWS: Split proxy from hub in PAWS - https://phabricator.wikimedia.org/T129208#2098091 (10yuvipanda) [06:06:09] 10PAWS: Setup an icinga check for PAWS - https://phabricator.wikimedia.org/T129209#2098209 (10yuvipanda) [10:19:26] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:19:40] PROBLEM - Puppet failure on tools-services-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:08] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:08] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:22] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:36] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:38] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:44] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:52] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:52] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:20:54] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:21:06] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:22:25] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:22:25] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:22:26] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:22:54] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:22:55] PROBLEM - Puppet failure on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:22:55] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:22:55] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:23:59] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:24:01] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:24:09] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:24:11] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:24:11] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:25:22] PROBLEM - Puppet failure on tools-exec-1221 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:38:05] creation of new instances in wikitech seems broken: I just created one, but can't log in. There's some ruby log spam on the console and the error message "puppet-agent[1358]: Did not receive certificate" [10:43:38] PROBLEM - SSH on tools-exec-1216 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:29] RECOVERY - SSH on tools-exec-1216 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [11:44:39] Whenever I'm trying to switch to visual editor, i'm getting this error. [11:44:39] 127.0.0.1:8080 says : [11:44:39] Error loading data from server: HTTP 503 [11:57:19] 6Labs, 10Datasets-General-or-Unknown: Labs dumps mirror directories in /public/dumps/public/ no longer world-readable - https://phabricator.wikimedia.org/T128945#2098582 (10ArielGlenn) I saw the same issue today after the run and manually corrected it. Investigating now. [12:34:56] anyone ? [12:35:04] https://www.irccloud.com/pastebin/lwdA8yvp/ [12:40:04] srijan, which wiki? [12:41:36] I didn't get, what do you mean by which wiki? Wikimedia? [12:43:47] oh, I suppose it is your own installation on labs? [12:44:43] if not, I was asking because there are 800 wikis on wikimedia [12:46:18] actually i have installed visual editor using vagrant. https://www.mediawiki.org/wiki/Extension:VisualEditor [12:46:24] but i'm unable to run it [12:46:38] i'm getting that error. [12:50:03] I am not familiar with visual editor, you may have maybe more lack on a #mediawiki-* channel such as #mediawiki-visualeditor [13:02:10] (03Abandoned) 10Youni Verciti: Add the public_html folder to easily update the html code [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/273320 (owner: 10Youni Verciti) [13:04:59] 10Tool-Labs-tools-Global-user-contributions, 7Easy, 7JavaScript: GUC counter doesn't increment - https://phabricator.wikimedia.org/T123879#2098632 (10Devirk) a:3Devirk [13:29:00] (03PS1) 10Youni Verciti: First commit on branch dev [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/275802 [13:30:48] (03CR) 10Youni Verciti: "Trying to work on dev's branch." [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/275802 (owner: 10Youni Verciti) [13:58:12] 6Labs, 10Datasets-General-or-Unknown, 13Patch-For-Review: Labs dumps mirror directories in /public/dumps/public/ no longer world-readable - https://phabricator.wikimedia.org/T128945#2098725 (10ArielGlenn) 5Open>3Resolved Ran by hand (still running) and it's finally doing the right thing. [14:09:32] 6Labs: tools-worker-1002 is un-sshable - https://phabricator.wikimedia.org/T129002#2098750 (10chasemp) 5Open>3Resolved a:3chasemp A few things: * Definitely unreachable and console said it had some stuck tasks but no other info * There doesn't seem to be the characteristic load issues we have seen with th... [14:33:58] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2098815 (10Krenair) Which key did you add to which user? On wikitech-static I have a separate (different from prod, different from labs) key for my own account... [14:55:02] (03PS2) 10Hashar: Introduce tox as an entry point [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265735 [14:55:30] (03PS3) 10Hashar: Introduce tox as an entry point [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265735 [14:56:46] (03CR) 10Hashar: "check experimental" [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/265735 (owner: 10Hashar) [15:01:03] yuvipanda: I need your help to review https://gerrit.wikimedia.org/r/#/c/274382/. I changed code in k8s module and I'm missing a lot of context to be confident about it. And I have no idea how to test it... [15:09:25] (03CR) 10Hashar: "check experimental" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/264926 (owner: 10Aude) [15:11:59] 10PAWS, 6Revision-Scoring-As-A-Service: Install revscoring inside PAWS - https://phabricator.wikimedia.org/T120317#2098959 (10Halfak) BTW, a PAWS demo of building and using a classifier model: https://github.com/wiki-ai/editquality/blob/master/ipython/reverted_detection_demo.ipynb [15:13:35] (03PS1) 10Hashar: build requires grunt-cli [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/275823 [15:13:55] (03CR) 10Hashar: "check experimental" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/275823 (owner: 10Hashar) [15:14:11] (03CR) 10Hashar: "Required as part of moving the npm job to disposable instance ( T119143 )" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/275823 (owner: 10Hashar) [15:15:14] (03CR) 10Hashar: [C: 032] build requires grunt-cli [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/275823 (owner: 10Hashar) [15:21:08] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [15:22:24] RECOVERY - Puppet failure on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [15:22:54] RECOVERY - Puppet failure on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [15:23:15] (03Merged) 10jenkins-bot: build requires grunt-cli [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/275823 (owner: 10Hashar) [15:23:58] RECOVERY - Puppet failure on tools-exec-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:38] RECOVERY - Puppet failure on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:27:25] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:11] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:28] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:58] RECOVERY - Puppet failure on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:03] RECOVERY - Puppet failure on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:05] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:53] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:18] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [15:40:38] RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [15:41:31] !log tools.stewardbots Modifyied .description [15:41:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [15:42:54] RECOVERY - Puppet failure on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:56] RECOVERY - Puppet failure on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:16] RECOVERY - Puppet failure on tools-exec-1213 is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:30] can sb help me using WinSCP to access my tools project? I'm stuck and instructions at wikitech are way outdated. [16:39:45] mafk: 'way outdated'? [16:40:08] valhallasw`cloud: I'd say so. Files are from 2013 [16:40:22] and I can't find those either in winscp as of today [16:41:00] I haven't updated winscp recently, but I'm unaware of any large changes [16:41:18] I'm also not sure which page you're looking at [16:41:50] https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP looks fine to me [16:41:59] https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP [16:42:09] that one yep [16:42:20] I can't manage to configure it to access [16:43:37] what did you do, and what happened instead? [16:44:38] valhallasw`cloud: yes, I downloaded WinSCP, and tried to configure it to access some instances I have access to [16:44:44] but it fails on authentication [16:44:53] I think I'm doing something wrong [16:45:08] mafk: instances or tools? [16:45:33] tools [16:45:46] I can't connect to tools, so I can't access the instances [16:46:59] mafk: instances are seperate VMs in labs, and that's probably not what you mean? [16:47:32] valhallasw`cloud: nevermind, resolved now [16:48:02] what was the issue? [16:53:41] mafk? [16:53:48] oh, sorry [16:53:54] something with SSH keys [16:54:02] ah, that explains [17:00:25] oh pine :/ [17:01:03] the http proxy for labs doesn't pass XFF headers does it? [17:01:28] bd808: I'm not sure, but it's trivial to get the information anyway (load an external resource) [17:01:50] right. welcome to tcp/ip networking [17:01:53] in any case, it would be a header thus not logged [17:02:09] I think that's the most important part (no storage of that information by default) [17:03:58] I want to respond to pine with an explanation of how the internet works but I'm going to resist that urge [17:05:54] bd808: novaproxy sets the XFF header [17:06:02] (modules/role/manifests/labs/novaproxy.pp) [17:07:25] I guess that's legit. A tool ot other labs project could be doing geolocation, etc that needs the UA's IP [17:07:54] tools doesn't -- it's only for the central labs proxy [17:18:50] (03PS1) 10MarcoAurelio: Bring back hat-web-tools [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275850 [17:35:45] bd808: what did pine do now? [17:36:46] (03PS2) 10MarcoAurelio: Bring back hat-web-tools [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275850 [17:36:52] harej: just me reacting to an email thread on labs-l "Labs privacy policy questions" [17:37:16] and not reacting very professionally. :/ I should be a better person [17:37:32] it's pine so you're okay [17:38:12] hi - uninterested third-party here.. I read that exchange about the pwds .. it seems a magnet for the insecure-minded participant [17:38:39] that is, it seemed to me that the labs responses were balanced [17:38:55] ideally, everything gets stronger from the exchange [17:41:21] (03PS3) 10MarcoAurelio: Bring back hat-web-tools [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275850 [17:42:21] 6Labs, 10Horizon: Horizon - Can't execute actions - https://phabricator.wikimedia.org/T127440#2043827 (10Krenair) Does it show now, @Luke081515? [17:42:56] (03CR) 10MarcoAurelio: [C: 032] Bring back hat-web-tools [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275850 (owner: 10MarcoAurelio) [17:44:22] 6Labs, 7Tracking: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2099584 (10Andrew) [17:44:48] 6Labs, 10Horizon, 7Tracking: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2099599 (10Krenair) [17:47:43] 10Wikibugs: Wikibugs links sometimes to the creation event, not to the mentioned comment - https://phabricator.wikimedia.org/T129246#2099604 (10Luke081515) [17:47:55] !log tools.stewardbots Merged https://gerrit.wikimedia.org/r/#/c/275850/ [17:47:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [17:56:16] 6Labs, 10Horizon: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2099667 (10Krenair) [17:57:54] 6Labs, 10Horizon: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2099584 (10Krenair) a:5Andrew>3Krenair [18:08:44] (03PS1) 10MarcoAurelio: HTML fixes for hat-web-tool/projects.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275861 [18:09:27] gehel: I probably won't be able to fully look at it this week, and am on vacation next week... :( sorry! [18:11:20] 6Labs: tools-worker-1002 is un-sshable - https://phabricator.wikimedia.org/T129002#2099733 (10yuvipanda) ok! The kernel downgrade did happen (from 4.2 to 3.19) though, so maybe the issue @faidon found also affects 3.19 kernels :( [18:12:24] bd808: valhallasw`cloud novaproxy did not send XFF to begin with [18:12:28] bd808: valhallasw`cloud and then the UTRS project wanted it [18:12:31] so we started sending it [18:12:41] because I didn't want to touch OSM code to make it configurable per project :| [18:12:59] lazypanda [18:13:02] ;) [18:14:03] :( [18:14:05] oh well [18:15:59] 6Labs: tools-worker-1002 is un-sshable - https://phabricator.wikimedia.org/T129002#2099754 (10yuvipanda) Also confirmed that the node has repooled itself :D [18:17:00] (03CR) 10MarcoAurelio: [C: 04-1] "Passed the HTML through an online editor which did some changes. Please see the inline comments. Until we are sure this will be working, d" (034 comments) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275861 (owner: 10MarcoAurelio) [18:17:22] yuvipanda: if letting XFF through or not was going to be a choice of each project I think just sending it is probably the right choice from a technical complexity POV [18:18:01] My security paranoia hat doesn't extend to IP addresses strangely enough [18:18:06] bd808: whitelisting would probabyl be fairly trivial [18:29:34] (03PS2) 10MarcoAurelio: HTML fixes for hat-web-tool tools on stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275861 (https://phabricator.wikimedia.org/T128743) [18:30:16] 6Labs, 10Tool-Labs, 6Operations, 10Traffic, and 2 others: Detect tools.wmflabs.org tools which are HTTP-only - https://phabricator.wikimedia.org/T128409#2099871 (10Dzahn) Yea, it meant the request came via http and was 200. [18:35:15] (03CR) 10MarcoAurelio: [C: 04-1] "Some questions on code. Needs review." (0312 comments) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275861 (https://phabricator.wikimedia.org/T128743) (owner: 10MarcoAurelio) [18:35:38] 6Labs, 10Horizon: Horizon - Can't execute actions - https://phabricator.wikimedia.org/T127440#2099902 (10Luke081515) I can't login at the moment because 2FA is required... I would prefere if 2FA is optional like at wikitech. [18:38:22] yuvipanda: Damn! Is there anyone else who has some idea on how our kubernetes works? [18:38:50] unfortunately not [18:39:21] I think trying to "prevent" the reveal of client ip's to internal systems in tools is a war of escalation [18:39:33] Ok, I'll raise the risk to whoever ... and we'll see how we move on this... [18:39:41] teh only semi sane things we coudl do would easily be overcome by the only people we are actually concerned about [18:41:47] gehel: what's the 'risk'? [18:41:59] (03CR) 10Luke081515: "I don't use this tool, and I don't read the whole code, but I hope I can help at the few commented places, my comments there don't depend " (036 comments) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275861 (https://phabricator.wikimedia.org/T128743) (owner: 10MarcoAurelio) [18:42:07] chasemp: yeah, I think the only sane thing we can do is preventing the IPs from being logged by default [18:42:26] which is basically the current situation [18:42:43] valhallasw`cloud: that makes sense [18:43:08] if people think their IP is private on the internet [18:43:27] they are going to be very disapointed [18:44:08] yuvipanda: that's a prerequisite for traffic encryption on elasticsearch and we'd like to have that in place for the codfw switchover. It's not a blocker, don't cancel your holidays for that ;-) [18:44:30] gehel: woah, how is k8s a prerequisite for traffic encryption on es?! [18:44:43] that made me curious as well [18:45:51] yuvipanda: there was a push to do some refactoring and use the same code to expose puppet certificates. But I can just rollout what I need and keep the k8s refactoring for later. [18:46:20] yuvipanda: It would have been nicer to do all at the same time and not have a 3rd way to expose those certs... [18:46:46] gehel: ah, right. but the underlying context - I provided some in that patch - is that the certs are in *different locations* based on wether you are on a self hosted puppetmaster or not [18:47:18] and so it practically needs https://phabricator.wikimedia.org/T120159 to be fixed [18:47:22] for it to be a viable solution [18:47:39] (this isn't too visible if you haven't really dealt with labs too much) [18:48:51] I did see a puppet_ssldir in wmflib which seems to abstract this particular issue... [18:49:31] not quite, it only bandaids it [18:49:44] because there's still two totally different certs and two totally different puppet modules being used [18:49:48] but yeah, I do not understand enough about the underlying issue to be confident to move forward on my own. So no sweat, I'll take the k8s part out of my patch and wait for you to be back from holiday [18:50:02] and I am abhorent to the bandaid since it affects me fairly disporportionately (since I work far more on labs :D) [18:50:05] gehel: cool :) [18:50:15] gehel: thanks for reaching out and attempting cleanups! that's highly appreciated :D [18:50:38] yuvipanda: thanks for the help anyway! [18:51:03] (03PS3) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) [18:51:59] gehel: np [19:04:50] (03PS1) 10Matanya: adding delete.php tool [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275886 [19:12:59] (03PS2) 10Matanya: adding delete.php tool [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275886 [19:13:50] (03PS3) 10Matanya: adding delete.php tool [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275886 [20:01:52] 6Labs: Migrate DNS entries from ldap to designate - https://phabricator.wikimedia.org/T129272#2100354 (10Andrew) [20:28:40] 6Labs, 10Horizon: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2100464 (10AlexMonk-WMF) a:5Krenair>3AlexMonk-WMF [20:30:08] yuvipanda: is the tools proxy redis replication one-way or two-way? [20:30:16] valhallasw`cloud: one way [20:32:21] * valhallasw`cloud ponders [20:37:57] (03PS2) 10ArthurPSmith: Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 [20:43:50] (03CR) 10ArthurPSmith: [C: 031] Updated nuclides charts to display using SVG; includes some refactoring into common components [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [20:44:14] valhallasw`cloud: yuvipanda what's the story on a forward proxy for labs instances [20:44:26] I remember the kerfuffle w/ it being abuse possible using the prod one [20:44:35] was taht blocked entirely or can it still be used but not dangeriously? [20:44:45] forward proxy? [20:45:07] :) just a normal web proxy for making requests [20:45:13] we firewalled that off [20:45:18] it was the proxy in production [20:45:21] that was accessible from labs [20:45:25] and shouldn't have been [20:45:27] so we killed it [20:45:30] and all was well [20:45:34] totally, but there is no alternative is that right? [20:45:38] nothing to do with the novaproxy [20:45:45] labs instances can just connect directly to the outside world [20:45:47] (NATted) [20:45:49] indeed [20:45:59] so no alternative needed, and we killed the known alternatives [20:46:04] ok tx, purely for my edification [20:46:37] a proxy would be useful in some limited senses, I know ppl wanted it for packaging or some thing, not sure what CI did [20:46:51] esp maybe for tools where it's possible we do many dupe requests for anyone following change feeds [20:46:58] I really don't know and don't want to dive into it :) just commentary [20:47:02] on the thinking [20:47:28] until bw is an issue I'm not too concerned [20:48:01] yeah [20:48:03] ^ [20:48:11] we can deal with it if network bw is a problem [20:50:28] (03CR) 10ArthurPSmith: "responses to comments on patchset 1 - I think everything is fixed!" (038 comments) [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/275521 (owner: 10ArthurPSmith) [20:55:15] The biggest need for a caching proxy is https://phabricator.wikimedia.org/T106452 (composer) [20:55:45] but the new composer versions do tls verification things that probably stop a proxy from being useful [20:56:04] ah, I know I have seen a few asks about it over the last few years [20:56:13] I just wasn't sure other than getting banned from prod :) what the outcome was [20:56:24] I'm not opposed...but it seems very low priority afacit [20:56:30] afaict? even [20:57:52] 6Labs: Migrate DNS entries from ldap to designate - https://phabricator.wikimedia.org/T129272#2100615 (10Andrew) [21:00:50] (03CR) 10MarcoAurelio: [C: 032] adding delete.php tool [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275886 (owner: 10Matanya) [21:10:02] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [21:13:23] (03Merged) 10jenkins-bot: adding delete.php tool [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275886 (owner: 10Matanya) [21:18:18] (03CR) 10MarcoAurelio: "Still not working at " [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275886 (owner: 10Matanya) [21:20:50] !log tools.stewardbot Merged https://gerrit.wikimedia.org/r/275886 [21:20:50] tools.stewardbot is not a valid project. [21:21:04] !log tools.stewardbots Merged https://gerrit.wikimedia.org/r/275886 [21:21:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [21:34:02] (03Abandoned) 10MarcoAurelio: HTML fixes for hat-web-tool tools on stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275861 (https://phabricator.wikimedia.org/T128743) (owner: 10MarcoAurelio) [21:50:02] (03CR) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots (031 comment) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [21:54:25] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2100854 (10Krenair) Am now in, after a bit of fiddling around with passwords and `sshd -d`, it turns out the issue was with file permissions on my authorized_k... [22:07:09] how will I know when my tool has been created, the help page suggests it will take a minute or two [22:07:52] TheDaveRoss_: you could try on teh bastion "become $toolname" or look in /data/project/$toolname [22:08:27] I have tried that a few times, for some reason that command is causing the ssh connection to become unresponsive [22:08:37] so I thought I would check if I am becoming something which doesn't exist [22:09:18] what is the tool name? [22:09:41] thedavetools [22:10:19] looks to be there and become works for me (tm) [22:10:22] not sure why that would be [22:10:40] are you on tools-bastion-05.tools.eqiad.wmflabs? [22:10:51] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: hat-web-tools import for stewardbots - https://phabricator.wikimedia.org/T128743#2100916 (10MarcoAurelio) [22:11:57] I am not sure where I am, I logged in to login.tools.wmflabs.org and then it says I am in /home/thedaveross/ [22:12:13] and when I send the command become thedavetools everything stops working [22:14:02] well, it worked in putty, thanks [22:24:01] Hi it seems that cache-rsync is down. [22:27:45] RECOVERY - Puppet failure on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [23:01:25] !log tools.stewardbots moved some files from old folders, miscellaneous fixes [23:01:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [23:08:17] !log tools.stewardbots chmod 770 to a folder [23:08:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL, Master [23:20:34] 6Labs, 10Monitoring, 13Patch-For-Review, 7Shinken, 7Upstream: shinken.wmflabs.org redirects on https-login to http - https://phabricator.wikimedia.org/T85326#2101391 (10Aklapper) Upstream ticket is closed and says "can you try webui2 instead ? and make a PR for the new webui2 if you encounter this issue... [23:42:58] 10Wikibugs: wikibugs linking to wrong action in IRC - https://phabricator.wikimedia.org/T129296#2101456 (10greg) [23:44:12] 10Wikibugs: wikibugs linking to wrong action in IRC - https://phabricator.wikimedia.org/T129296#2101472 (10greg) [23:47:16] 10PAWS, 6Research-and-Data: Build a pool of beta testers for PAWS - https://phabricator.wikimedia.org/T129297#2101483 (10DarTar) [23:47:56] 10Wikibugs: Wikibugs links sometimes to the creation event, not to the mentioned comment - https://phabricator.wikimedia.org/T129246#2099604 (10Legoktm) I guess the screenscraping finally broke? [23:48:35] 10Wikibugs: Wikibugs links sometimes to the creation event, not to the mentioned comment - https://phabricator.wikimedia.org/T129246#2099604 (10mmodell) does it really use screenscraping? [23:50:18] 10Wikibugs: Wikibugs links sometimes to the creation event, not to the mentioned comment - https://phabricator.wikimedia.org/T129246#2101504 (10Legoktm) Yes, see {T126715}.