[01:17:47] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0] [02:27:45] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [06:53:24] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:28:26] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [08:21:59] 6Labs, 10Tool-Labs, 10Dumps-Generation: /public/dumps/public/ is not updating on Tool Labs - https://phabricator.wikimedia.org/T115969#1752413 (10ArielGlenn) [08:45:41] 6Labs, 10Tool-Labs, 10Dumps-Generation: /public/dumps/public/ is not updating on Tool Labs - https://phabricator.wikimedia.org/T115969#1752425 (10ArielGlenn) I still don't see any new dumps copied over. Investigating. [09:28:20] 6Labs, 10Tool-Labs, 10Dumps-Generation: /public/dumps/public/ is not updating on Tool Labs - https://phabricator.wikimedia.org/T115969#1752463 (10ArielGlenn) 5Open>3Resolved a:3ArielGlenn A casualty of the rsync version on trusty, fixed in https://gerrit.wikimedia.org/r/#/c/248824/ It was putting comm... [09:46:37] PROBLEM - Puppet staleness on tools-k8s-bastion-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [12:34:35] 6Labs, 10Tool-Labs: Install Perl module Sort::Fields - https://phabricator.wikimedia.org/T116579#1752793 (10Stigmj) 3NEW [13:09:26] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install Perl module Sort::Fields - https://phabricator.wikimedia.org/T116579#1752848 (10coren) 5Open>3Resolved a:3coren [13:18:59] (03CR) 10Alex Monk: "Yuvi: Ping" [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/245200 (https://phabricator.wikimedia.org/T69927) (owner: 10Alex Monk) [13:29:12] anyone know how long it takes for a newly installed perl module to be available in the grid? (Perl module Sort::Fields) [15:28:02] 6Labs, 10Labs-Infrastructure, 6operations, 7Monitoring, 3labs-sprint-118: monitor expiration of labvirt-star SSL cert - https://phabricator.wikimedia.org/T116332#1753375 (10Andrew) [15:30:11] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad, 3labs-sprint-118: Rack/Setpup labvirt1010 and 1011 - https://phabricator.wikimedia.org/T116019#1753384 (10Andrew) [15:34:59] 6Labs, 3labs-sprint-118: Document support levels for tools and labs projects - https://phabricator.wikimedia.org/T116598#1753409 (10Andrew) [16:00:47] Stigmj: about 20mins [16:03:59] 6Labs, 6Discovery, 7Elasticsearch, 3labs-sprint-116, and 2 others: Replicate production elasticsearch indices to labs - https://phabricator.wikimedia.org/T109715#1753583 (10yuvipanda) [16:04:12] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1753584 (10yuvipanda) [16:04:23] 6Labs, 10Tool-Labs, 10Incident-20150617-LabsNFSOutage, 3labs-sprint-117, 3labs-sprint-118: Re-enable cron for tools on tool labs - https://phabricator.wikimedia.org/T104614#1753585 (10yuvipanda) [16:05:51] 6Labs, 10Tool-Labs, 3labs-sprint-118: Enforce that containers from a user run with the uid assigned to that user - https://phabricator.wikimedia.org/T116504#1753594 (10yuvipanda) [16:06:13] 6Labs, 10Tool-Labs, 3Labs-Sprint-115, 3labs-sprint-116, 3labs-sprint-118: Write admission controller disabling mounting of unauthorized volumes - https://phabricator.wikimedia.org/T112718#1753596 (10yuvipanda) [16:16:59] 6Labs, 6Phabricator, 7Puppet: phabricator puppet at labs broken - https://phabricator.wikimedia.org/T116442#1753651 (10Dzahn) Why can't we use the same class in prod and labs? That's the idea of testing changes, [16:23:05] 6Labs, 10Incident-20141106-Labstore: Ensure that opsen are paged on failure of labstore1001's NFS service - https://phabricator.wikimedia.org/T76402#1753682 (10yuvipanda) 5Open>3Resolved So I have icinga hitting tools-checker.wmflabs.org/nfs/home and alerting, and it's alerted us a couple of times when nee... [16:23:19] 6Labs, 10Incident-20141106-Labstore: Ensure that opsen are paged on failure of labstore1001's NFS service - https://phabricator.wikimedia.org/T76402#1753684 (10yuvipanda) a:5coren>3yuvipanda [16:23:41] 6Labs, 10Tool-Labs, 10Labs-Team-Backlog: Fix Labs' PAM config mess - https://phabricator.wikimedia.org/T85910#1753685 (10yuvipanda) [16:23:59] 6Labs, 10Labs-Team-Backlog, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, and 4 others: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1753686 (10yuvipanda) [16:24:38] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, 3labs-sprint-117: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1753690 (10Andrew) 3NEW a:3Andrew [16:25:18] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, and 2 others: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1753690 (10Andrew) [16:26:16] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, and 2 others: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1753700 (10RobH) I was chatting about this with Andrew. So since all mgmt is on 'dumb' switches, we don't support multiple mgmt vlans unless... [16:30:20] 6Labs, 10Labs-Team-Backlog, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Switch NFS server back to labstore1001 - https://phabricator.wikimedia.org/T107038#1753717 (10coren) @cmjohnson: We really want to do this early this week rather than later - can you set aside a 2h period in the DC for this... [16:30:25] 6Labs, 10Labs-Infrastructure: Change network configuration to network-node per compute-node - https://phabricator.wikimedia.org/T48819#1753719 (10chasemp) 5Open>3Resolved a:3chasemp We decided not to do this. We pursued a manual failover for current nova-network labnet hosts and are looking at pursuing N... [16:31:25] Coren: need to provide user notice. [16:31:55] YuviPanda: That requires a known time and date. I really really don't want to delay this another week given how flaky things have been this weekend. [16:32:02] 6Labs, 10Labs-Infrastructure: Labs needs a reliable and communicative status dashboard - https://phabricator.wikimedia.org/T95922#1753723 (10chasemp) p:5Triage>3Normal [16:32:48] 6Labs, 10Labs-Infrastructure: Change network configuration to network-node per compute-node - https://phabricator.wikimedia.org/T48819#1753734 (10chasemp) [16:32:49] 6Labs, 10Labs-Infrastructure: Add BGP support to Quantum for floating IP addresses - https://phabricator.wikimedia.org/T48821#1753731 (10chasemp) 5Open>3Invalid a:3chasemp We are trying to move away from nova-network and Grizzly mentioned here is legacy [16:32:53] Coren: sure, but at least one or two days sounds prudent. And I'm sure there's enough work to do for a few days. So make sure to provide user notice as soon as we know a date and time [16:33:27] YuviPanda: Agreed. As soon as chris has a window set, I'll email the lists. [16:34:46] 6Labs, 10Labs-Infrastructure, 7Monitoring: ganglia for virt cluster shows all the wrong things - https://phabricator.wikimedia.org/T101043#1753745 (10chasemp) 5Open>3Invalid a:3chasemp >>! In T101043#1343748, @Andrew wrote: > Right at this moment it is working properly again. As far as I know no one d... [16:37:36] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, and 2 others: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1753783 (10chasemp) If the idea is these physical boxes are totally under the control of the relevant project admins we should consider mimic... [16:38:45] 6Labs, 10Labs-Team-Backlog, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Switch NFS server back to labstore1001 - https://phabricator.wikimedia.org/T107038#1753796 (10Cmjohnson) Let's plan on Wednesday morning at 930 Eastern. [16:48:56] 6Labs, 10Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#1753832 (10chasemp) is this still an ongoing blocker for people? [16:51:25] Hi YuviPanda. I've started working on the survey results. I'm hoping that we have all the results ready today. Will ping when I have more. [16:53:00] leila: awesome [16:58:25] Hey! A bunch of jobs in my tool have erred out with error message "_mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query')". Anything changed in the backend? Enforced time limit or something? These jobs took an hour or two to run usually, sometimes more. [17:08:30] 6Labs, 10Labs-Infrastructure, 7Monitoring: ganglia for virt cluster shows all the wrong things - https://phabricator.wikimedia.org/T101043#1753927 (10Legoktm) 5Invalid>3Resolved [17:08:48] 6Labs, 6Scrum-of-Scrums: Increase cpu and disk quota for the 'search' group - https://phabricator.wikimedia.org/T116292#1753929 (10EBernhardson) [17:13:04] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, and 2 others: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1753968 (10Andrew) [17:13:47] 6Labs, 3Labs-Sprint-109, 7Monitoring, 5Patch-For-Review, and 3 others: Monitor nova services - https://phabricator.wikimedia.org/T90784#1753975 (10Andrew) [17:14:10] 6Labs: Support bare-metal server allocation in labs - https://phabricator.wikimedia.org/T95185#1753977 (10chasemp) p:5Triage>3Normal [17:15:46] hi, can one of the admins try to login to mlp.math.eqiad.wmflabs I get Server refused our key. even though I recreated this instance [17:17:42] ... it is also not visible on https://tools.wmflabs.org/nagf/?project=math#h_mlp [17:18:09] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, and 2 others: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1754006 (10Andrew) a:5Andrew>3None [17:18:11] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, and 2 others: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#1754008 (10Andrew) a:5Andrew>3None [17:24:24] 6Labs, 10Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#1754049 (10Dereckson) Quick poll on IRC wikimedia-commons show it's still needed, but not with an high priority. [17:26:53] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-eqiad, 3labs-sprint-118: Rack/Setpup labvirt1010 and 1011 - https://phabricator.wikimedia.org/T116019#1754054 (10chasemp) a:5Cmjohnson>3chasemp [17:28:34] 6Labs, 10Labs-Team-Backlog: Support bare-metal server allocation in labs - https://phabricator.wikimedia.org/T95185#1754084 (10mark) [17:34:00] 6Labs, 10Labs-Team-Backlog: Support bare-metal server allocation in labs - https://phabricator.wikimedia.org/T95185#1754107 (10yuvipanda) mwoffliner as it currently is is a good test case, I think. I've poked Kelson on IRC and will update. [17:34:29] YuviPanda or andrewbogott,, could you guys please look at https://gerrit.wikimedia.org/r/#/c/248374/ ? [17:36:45] MaxSem: you can cherrypick it to beta puppetmaster and put it for puppetswat? :D [17:37:50] speaking of puppetswat, can you +1 https://gerrit.wikimedia.org/r/#/c/248622/ YuviPanda? [17:38:15] I added it to my list of things to go in puppetswat [17:38:34] Krenair: done [17:39:44] I'm hoping andrewbogott can do https://gerrit.wikimedia.org/r/#/c/248619/ and _joe_ can do https://gerrit.wikimedia.org/r/#/c/248634/ [17:39:51] and hopefully every other op can do everything else on the list [17:42:41] err, beta is not running off puppet tip? [17:43:06] 6Labs, 10Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#1754139 (10Nemo_bis) Maybe we should try ownCloud. :D [17:45:17] Krenair, you're occupying most of puppet swat for yourself! :P [17:45:32] MaxSem, hmm... should be [17:45:53] last proper commit it has is "Date: Mon Oct 26 10:35:10 2015 -0700", then a load of local hacks [17:46:03] Which is from 10 minutes ago [17:46:16] and yeah, I've filled 7 of the 8 spaces of puppetswat already [17:46:27] 6Labs, 10Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#1754156 (10Dzahn) >>! In T33828#1754139, @Nemo_bis wrote: > Maybe we should try ownCloud. :D I had the same thought. owncloud is free software a... [17:48:08] 6Labs, 10Labs-Infrastructure, 6operations, 3labs-sprint-117: Allocate labs subnet in dallas - https://phabricator.wikimedia.org/T115491#1754167 (10mark) For the Labs hosts/support vlans we can just follow the eqiad model for now, and copy that to codfw, with similar IP allocations as well. [17:49:47] 6Labs, 10Labs-Infrastructure, 6operations, 3labs-sprint-117: Allocate subnet for labs test cluster instances - https://phabricator.wikimedia.org/T115492#1754169 (10mark) Let's allocate a similar amount of IPs (/20 iirc or thereabouts) as we did in eqiad, but experiment with how to use them with Neutron. I... [17:57:39] 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: adding/removing project members updates project page, changing projectadmins does not - https://phabricator.wikimedia.org/T73164#1754224 (10Krenair) 5Open>3Resolved Hopefully this will work now - should arrive on wikitech on Wednesday as part of t... [18:07:11] Niharika: hey! I think idle connections die if left open for too long [18:07:32] physikerwelt: checking [18:07:57] physikerwelt: hmm, denies root key too. did you delete and re-create it within a few minutes with the same name? [18:20:08] MaxSem, Andrew dealt with one of them already, you can have another slot now :P [18:20:24] meh, I still hate you :P [18:20:58] 10Quarry, 7Database: Unpatrolled edits don't work on plwiki - https://phabricator.wikimedia.org/T116631#1754331 (10The_Polish) 3NEW [18:31:14] 10Quarry, 7Database: Unpatrolled edits don't work on plwiki - https://phabricator.wikimedia.org/T116631#1754401 (10Krenair) What? `select sum(rc_patrolled = 1), sum(rc_patrolled = 0) from plwiki_p.recentchanges;` shows some patrolled and some unpatrolled RCs. [18:31:24] 10Quarry, 7Database: Unpatrolled edits don't work on plwiki - https://phabricator.wikimedia.org/T116631#1754402 (10Krenair) I very much doubt this is an issue with Quarry itself. [18:38:42] 6Labs, 10Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#1754451 (10chasemp) p:5Triage>3Low [18:54:49] YuviPanda: Hi there! Umm, the query was running fine a few weeks. Then it suddenly stopped working. I dunno what idle means here though, it was a complicated query. [18:58:14] Niharika: https://phabricator.wikimedia.org/T76699 seems related [19:05:42] legoktm: can you try sshing into tools-dev.wmflabs.org? [19:05:47] has new ssh version... [19:05:56] MaxSem: ^ you too (assuming you're on Windows?) [19:07:38] Oh, well. YuviPanda: What would you suggest I do here? Try and optimize it better? [19:08:11] Niharika: I'm not particularly sure. that's one option, and I don't know what exactly 'idle' means - perhaps it just means no data is passing through in either direction [19:08:27] and the reason it worked before is perhaps because some data was coming back all the time [19:08:31] instead of it just waiting there for hours [19:08:32] stepping back [19:08:38] YuviPanda: That would happen when a query is being processed, right? [19:08:39] why are you running multi hour queries on labsdb? [19:09:14] YuviPanda: https://en.wikipedia.org/wiki/Wikipedia:Database_reports [19:09:47] ah that [19:10:40] Niharika: I don't really have an answer for you, unfortunately. my mysql knowledge is severly limited [19:10:42] YuviPanda: did the fingerprint change? [19:10:49] legoktm: no [19:11:00] @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [19:11:00] @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ [19:11:00] @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ [19:11:13] YuviPanda: Okay, thanks anyway! [19:11:27] ECDSA key fingerprint is SHA256:OfgR6GTw8ObBQ1LbS+6NBVik1eEXrpSUvRkKOueUnQc. [19:11:27] ECDSA key fingerprint is MD5:80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57. [19:11:41] YuviPanda: I logged in [19:11:51] to tools-bastion-02 though??? [19:11:56] legoktm: hmm, that's interesting. what's your client ssh version? [19:12:39] km@km-tp ~> ssh -V [19:12:40] OpenSSH_6.9p1, OpenSSL 1.0.1k-fips 8 Jan 2015 [19:13:06] legoktm: yeah, bastion-02 is tools-dev [19:14:30] weird. [19:14:58] that's pretty not great and shouldn't happen? [19:15:25] chasemp: which bit? [19:15:28] the key changing? [19:15:42] right [19:16:01] yeah, my suspiciaon is that legoktm's client doesn't use ECDSA otherwise to older version of openssh [19:16:11] and then is freaking out over old cached version of key for newer version of ssh [19:16:14] I've no proof of that ofc [19:16:28] might as well have had openssh regenerate the keys when it reinstalled [19:17:46] looks like openssh regenerated them [19:18:57] huh no [19:19:10] legoktm: wait when was the last time you ssh'd into tools-dev? [19:19:22] legoktm: since those keys match what's on the host and on my local machine [19:19:34] and these changed many months ago [19:33:44] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations, and 2 others: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1754657 (10chasemp) Notes from meeting: Ironic will have its own model, for now the mgmt interface for any labs "hardware" node will be on i... [19:34:44] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-114: Ironic on Labs - https://phabricator.wikimedia.org/T110556#1754662 (10chasemp) [19:34:46] 6Labs, 10Labs-Team-Backlog: Support bare-metal server allocation in labs - https://phabricator.wikimedia.org/T95185#1754661 (10chasemp) [19:35:49] 6Labs, 10Labs-Team-Backlog: Support bare-metal server allocation in labs -- bootstrap mode - https://phabricator.wikimedia.org/T95185#1754665 (10chasemp) [19:42:58] Reedy: are you on windows? do you use putty? :D [19:43:10] No and no atm... [19:43:17] I can go get my windows laptop if needed [19:43:55] Reedy: what do you use for ssh? [19:44:02] PuTTy on windows [19:44:06] But I'm on mac [19:44:08] Reedy: I have a new version of sshd on tools-dev and need to test it on windows [19:44:12] Reedy: ah, what version of OS X? [19:44:17] 10.11 [19:44:23] ah damn [19:44:31] Reedy: yes, would be awesome if you can test on your windows laptop [19:44:33] I can go turn my windows laptop on no problem [19:49:59] Sooo.. Windows 7 x64 SP1. PuTTY 0.64 [19:50:05] legoktm: What do you want me to try? [19:50:29] Reedy: just ssh to tools-dev.wmflabs.org [19:50:30] uh [19:50:35] heh [19:50:56] latest release is 0.65 [19:51:01] will update in a few too [19:51:21] Reedy: yeah, but try on your current release first :D [19:53:29] YuviPanda: I probably last logged in over a year ago [19:53:47] legoktm: yeah, the ssh fingerprint definitely changed since then :D [19:54:31] And my wifi is sucking [20:00:25] YuviPanda: seems to be WFM [20:00:33] Reedy: both tools-login and tools-dev? [20:00:40] awesome [20:00:43] I just saw the entry in tools-dev [20:00:50] Reedy: this is 0.64 right? [20:01:06] 2015-10-26 20:00:51 We claim version: SSH-2.0-PuTTY_Release_0.64 [20:01:27] cool [20:01:35] now I Just need to find someone on an older version of OS X [20:01:37] No excuse for not upgrading it [20:01:46] heh [20:03:06] 0.65 is good too [20:03:35] I just realised the difference in resolution between my X220 and my rmbp 13" [20:05:37] 1366x768 on the x220 [20:06:28] oh wow [20:12:42] YuviPanda: I have Yosemite at home that I can test with later [20:13:14] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1754849 (10yuvipanda) So the new ssh version is on tools-bastion-02, and so far tested with the ssh versions on debian jessie, OS X 10.11, Putty 0.64 and 0.65... [20:30:13] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1754903 (10yuvipanda) Ok, verified on OS X 10.9 too. I'll update ssh on tools-login too [20:43:30] Negative24: Ping ;) [20:53:58] !log tools updated 6.9 ssh backport to all trusty hosts [20:54:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:54:12] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1755077 (10yuvipanda) New packages in aptly now. [20:57:35] a silly question: why if I run `curl http://localhost/ -H 'Host:www.wikipedia.beta.wmflabs.org'` on beta appservers I get domain not configured? [20:58:09] I think I've run into that before [20:59:39] MaxSem, try specifying the instance hostname instead of localhost [20:59:49] o_0 [20:59:54] It'll resolve to the internal IP instead of 127.0.0.1 [21:00:16] Which I guess apache just isn't listening on correctly somehow [21:00:25] thanks:) [22:50:10] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1755615 (10scfc) I can no longer `ssh` from `tools-bastion-01` to a Trusty instance with HBA: ``` scfc@tools-bastion-01:~$ ssh tools-exec-1201 true scfc@tools... [22:56:08] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1755639 (10yuvipanda) Interesting... ```Oct 26 22:55:34 tools-exec-1401 sshd[6690]: Rhosts authentication refused for yuvipanda: bad ownership or modes for ho... [22:57:46] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1755652 (10yuvipanda) With that fixed... ```Oct 26 22:57:13 tools-exec-1401 sshd[6956]: Failed hostbased for yuvipanda from 10.68.17.228 port 47679 ssh2: ECDS... [23:01:23] 6Labs: HBA failing to trust instances on tool labs - https://phabricator.wikimedia.org/T116687#1755679 (10yuvipanda) 3NEW [23:01:47] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-116, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#1681701 (10yuvipanda) @scfc I created T116687 to track that. [23:02:37] 6Labs: HBA failing to trusty instances on tool labs - https://phabricator.wikimedia.org/T116687#1755703 (10yuvipanda) [23:09:58] YuviPanda, hey [23:10:55] Krenair: hey [23:11:01] Can I get a public IP on the visualeditor project? [23:11:13] wanna open up restbase to demonstrate something [23:13:46] Krenair: hmm, on the visualeditor project? [23:13:50] yes [23:13:51] Krenair: why not use the proxy? [23:13:54] isn't it just htto? [23:13:57] *http [23:14:10] Can I open up port 7231 externally through that? [23:14:18] I guess I could give it a separate domain and put it on external port 80 [23:14:21] ugh, ok [23:14:22] that'll work [23:14:30] Krenair: you can specify port number in the proxy [23:14:38] I know I can do the internal port number, sure [23:14:39] but yeah, to the outside world it'll be port 80/443 [23:14:56] if absolutely necessary I can give you a public IP but would prefer to not :) [23:15:25] no I think I can manage it without [23:15:52] cool [23:15:54] thanks [23:34:43] krenair@visualeditor-switch:/srv/mediawiki-vagrant$ dig visualeditor-switch.wmflabs.org @labs-ns0.wikimedia.org | grep "IN A" [23:34:43] visualeditor-switch.wmflabs.org. 3600 IN A 208.80.155.156 [23:34:43] krenair@visualeditor-switch:/srv/mediawiki-vagrant$ dig visualeditor-switch.wmflabs.org @8.8.8.8 | grep "IN A" [23:34:43] visualeditor-switch.wmflabs.org. 3599 IN A 208.80.155.156 [23:34:46] YuviPanda, is that expected? [23:35:11] (the change in number) [23:37:16] Krenair: the change form "3600" to "3599" ? That's the TTL of the record... so 8.8.8.8 is caching it now for 3599 more seconds before refreshing from the authoritative NS. [23:37:26] Krenair: yeah what Stigmj said [23:38:58] 6Labs: "library-upgrader" labs project request - https://phabricator.wikimedia.org/T116697#1755830 (10Legoktm) 3NEW [23:41:10] legoktm, not paladox? [23:41:17] Krenair: ;) [23:41:27] not paladox yet? [23:41:46] hopefully it'll make his life easier too [23:42:11] but mainly I want to stop submitting patches against my account, and I thought I could improve all of it at the same itme [23:42:15] time* [23:59:44] YuviPanda, I got it working that way