[00:10:34] bd808: around? [00:11:12] Negative24: yeah, what's up? [00:11:51] bd808: have you done anything with docker-dev? [00:12:30] nope. my only docker playing around was with that mw-vagrant config [00:12:53] ok. seems its stopped responding with ssh. I'm going to recreate [00:13:58] oh, your host -- gotcha [00:14:06] nm. reboot seems to have woken it [00:14:11] cool [00:14:13] yeah [00:15:07] there were a few hosts that I had to kick in the last couple of week because they stopped talking to ldap correctly [00:16:37] now its ldap? [00:19:35] the ssh access relies on ldap. I saw at lest 3 hosts that had an nslcd (ldap cache) failure that kept logins by non-root folks from working last week. [00:20:00] not sure if there was a root cause or just an observed cluster of normal failures [00:20:56] hmm [00:21:08] and reboot resolved it? [00:21:51] yeah. that restarts the cache service (along with everything else) [00:22:39] if you could get access to the host just restarting nslcd would fix it too but that takes either a salt server or a root key once it goes haywire [00:23:51] yeah I don't have salt [00:26:28] let me know the hostname and i can restart it [00:26:50] mutante: a reboot resolved it. thanks though [00:27:04] okay [00:45:56] 6Labs, 6operations, 5Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#1618912 (10Krenair) @RobH: Do you think we should turn this into some sort of regular check? Not sure icinga necessarily would be appropriate, but... [00:49:02] 6Labs, 6operations, 5Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#1618923 (10Dzahn) @Krenair I think in a perfect world it would be detected by jenkins. If somebody touches a file in admin and there is a key in it it would have to compare it to all exist... [00:50:45] 6Labs, 6operations, 5Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#1618924 (10Krenair) >>! In T108078#1618923, @Dzahn wrote: > @Krenair I think in a perfect world it would be detected by jenkins. If somebody touches a file in admin and there is a key in i... [00:53:02] 6Labs, 6operations, 5Patch-For-Review: audit labs versus production ssh keys - https://phabricator.wikimedia.org/T108078#1618928 (10Krenair) That said, I think jenkins might be less annoying in operations/puppet than it is where I'm used to it - i.e. VE-MW (where you can't merge anything without being jenkin... [01:05:32] 6Labs, 6operations: labstore1002 not mounting all LVs after reboot - https://phabricator.wikimedia.org/T110832#1618943 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Sorted now. [01:05:34] 6Labs, 10Tool-Labs, 7user-notice: No file system on toollabs, unable to login, web service broken - https://phabricator.wikimedia.org/T110827#1618946 (10yuvipanda) [01:28:23] 6Labs, 10Tool-Labs: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#1618964 (10yuvipanda) 3NEW [01:34:32] 6Labs, 10Tool-Labs: Create debian packages for kubernetes - https://phabricator.wikimedia.org/T111888#1618990 (10yuvipanda) 3NEW [01:35:40] 6Labs, 10Tool-Labs: Create debian packages for kubernetes - https://phabricator.wikimedia.org/T111888#1618990 (10yuvipanda) Ok, so there is a flannel package in operations/debs/flannel (github.com/wikimedia/operations-debs-flannel) and kubernetes package in operations/debs/kubernetes (github.com/wikimedia/oper... [02:57:59] 6Labs, 10Tool-Labs: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#1619229 (10yuvipanda) I've an etcd + k8s master setup on tools-k8s-master-01 and worker nodes on tools-worker-01 and tools-worker-02 running \o/. Flannel is hitting etcd over https, and all... [02:59:17] 6Labs, 10Tool-Labs: Make sure that docker0 bridge comes up after flannel network is established - https://phabricator.wikimedia.org/T111893#1619230 (10yuvipanda) 3NEW [06:13:58] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1619382 (10ArielGlenn) wrong versions have been fixed. now dealing with bad minion count. [06:34:22] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1619404 (10MoritzMuehlenhoff) >>! In T99213#1616465, @ArielGlenn wrote: > I have a lame script for this stuff, well a few, and I should stick them somewhere others can steal pieces... [06:41:31] RECOVERY - Free space - all mounts on tools-exec-1206 is OK All targets OK [07:09:02] 6Labs, 10Tool-Labs: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#1619510 (10yuvipanda) [07:22:26] 6Labs, 10Tool-Labs: toolserver redirects double encode params - https://phabricator.wikimedia.org/T111839#1619567 (10valhallasw) > IE: http:// > becomes: > http%3A%2F%2 Those are equivalent (: = %2A, / = %2F). So: > ?q=http://en.wikipedia.org is equivalent to > ?q=http%3A%2F%2Fen.wikipedia.org but > ?q=ht... [07:31:21] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1619612 (10yuvipanda) 3NEW [07:32:39] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1619612 (10yuvipanda) So I'm passing --token-auth-file and it is pointing to a file in the valid format, but there's no acknowledgement from kubernetes that it even saw this. Unauthenticated calls suc... [07:33:40] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1619626 (10yuvipanda) (https://github.com/wikimedia/operations-puppet/blob/production/modules/k8s/templates/initscripts/kube-apiserver.systemd.erb for the commandline used) [07:38:50] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1619658 (10yuvipanda) Had same problem with the basic auth provider as well. [07:41:31] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1619661 (10yuvipanda) there is an ABAC policy file restricting the one account I have to only one namespace, but that also seems to have no effect. [08:17:06] 6Labs, 10Tool-Labs: Setup DNS for kubernetes services - https://phabricator.wikimedia.org/T111914#1619768 (10yuvipanda) 3NEW [08:21:07] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1619779 (10ArielGlenn) bad minion count instances are done with one exception, wbdocs.scrumbugz.eqiad.wmflabs which hangs on ssh. scripting up the key regeneration now. Note tha... [08:27:11] 6Labs, 10Tool-Labs: Setup DNS for kubernetes services - https://phabricator.wikimedia.org/T111914#1619799 (10yuvipanda) Our test cluster's DNS ran into https://github.com/kubernetes/kubernetes/issues/2996 - let's see how this one fares! [08:27:28] 6Labs, 10Tool-Labs: Setup DNS for kubernetes services - https://phabricator.wikimedia.org/T111914#1619801 (10yuvipanda) Also need to decide if we're ok using their pre-built images or want to build our own. [08:29:08] 6Labs, 10Tool-Labs: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#1619802 (10yuvipanda) [08:29:57] 6Labs, 10Tool-Labs: Add support to dynamicproxy for kubernetes based web services - https://phabricator.wikimedia.org/T111916#1619805 (10yuvipanda) 3NEW [08:33:17] 6Labs, 10Tool-Labs: Setup DNS for kubernetes services - https://phabricator.wikimedia.org/T111914#1619813 (10yuvipanda) Since the primary use case for this is T111916, and there are alternatives there, we could skip this for the initial setup. [09:28:51] 6Labs, 10Deployment-Systems, 10Labs-Infrastructure, 10Salt, 5Continuous-Integration-Scaling: Can not use git-deploy from tin.eqiad.wmnet to labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T111925#1619993 (10hashar) 3NEW [10:39:22] 6Labs, 10Tool-Labs: toolserver redirects double encode params - https://phabricator.wikimedia.org/T111839#1620140 (10Betacommand) https://tools.wmflabs.org/betacommand-dev/cgi-bin/cite.py?url=https%253A%252F%252Fwww.google.com%252F is the resulting address after redirection. [10:56:58] Is it possible to tell from the tool labs database replicas whether a user is blocked? Or is that one of the redacted things? [11:34:59] 6Labs, 10Tool-Labs: Error not reported with SVG Check - https://phabricator.wikimedia.org/T111939#1620345 (10Ruthven) 3NEW a:3Jarry1250 [11:48:44] 6Labs, 10Salt, 6operations: salt does not run reliably for toollabs / labs generally - https://phabricator.wikimedia.org/T99213#1620396 (10ArielGlenn) key regen completed, did another round of deleting keys of deleted instances, now checking on minions with authentication errors in the logs. [12:15:41] 6Labs, 10Tool-Labs, 7HTTPS: toolserver redirects double encode params - https://phabricator.wikimedia.org/T111839#1620444 (10valhallasw) This seems to be related to the http->https redirect on toolserver.org rather than the toolserver.org -> tools.wmflabs.org redirect: ``` valhallasw@maeglin:~$ curl -I -L "h... [12:26:28] 10Tool-Labs-tools-Other: Error not reported with SVG Check - https://phabricator.wikimedia.org/T111939#1620456 (10valhallasw) [13:18:01] 6Labs, 10Tool-Labs, 7HTTPS, 5Patch-For-Review: toolserver redirects double encode params - https://phabricator.wikimedia.org/T111839#1620556 (10valhallasw) 5Open>3Resolved a:3valhallasw ``` valhallasw@maeglin:sites$ curl -I -L "http://toolserver.org/~betacommand/cgi-bin/cite.py?url=http%3A%2F%2Fen.wi... [13:22:57] GoldenRing: fairly easy [13:41:53] 6Labs, 6operations, 5Patch-For-Review: labs salt master on jessie fails to install salt-master - https://phabricator.wikimedia.org/T110032#1620623 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi this works now, newly provisioned jessie below, thanks @Andrew! ```lines=15 filippo@test-cassandra3:~$ apt-cac... [13:43:33] 6Labs, 3labs-sprint-113: Separate scratch and tools NFS volumes to separate physical devices - https://phabricator.wikimedia.org/T111802#1620626 (10coren) a:3coren This is started on labstore1002 with `pvmove -n scratch /dev/md123 /dev/md125` At the current apparent rate, the operation should take between 5... [14:28:26] Hi [14:28:54] I have an issue with my server: osmit-cruncher1.eqiad.wmflabs [14:29:16] try to log in, server key refused [14:29:40] I can log in in another server without problem [14:30:08] Restarted the server, but doesn't work [14:30:41] I think that my server doesn't mount the partition with all the public keys [14:31:26] Someone please can check this? [14:33:25] It's not the first time that I see a strange behavior [14:36:37] 6Labs, 10Labs-Infrastructure, 3Labs-Sprint-111: Labs virt capacity expansion - https://phabricator.wikimedia.org/T107624#1620749 (10Andrew) [14:36:43] sbiribizio: That's not how key distribution has worked for some time; lemme go take a look. [14:37:17] Thanks [14:41:35] sbiribizio: It looks like it had been a *very* long time since puppet ran on that instance, and much of the configuration was woefully outdated. [14:42:28] I've just restarted the salt master, nothing changed on it, and it was out for at most one second, but there's notification anyways [14:43:34] sbiribizio: Try again? [14:44:06] OK, It works! [14:44:14] Thank you so much [14:44:18] No worries. [14:46:16] I can't seee where puppet is running on my server [14:47:04] sbiribizio: It's normally a cron job. [14:47:58] Nothing on cron root [14:48:20] crontab -l for the root user is empty [14:49:15] There is "2,32 * * * * root /usr/local/sbin/puppet-run > /dev/null 2>&1" in /etc/cron.d/puppet [14:49:22] So it should work, in theory. [14:50:37] Aha! [14:51:07] I see what your issue is: puppet-run tried to do an apt-get update first, but your sources.d points to broken repos. [14:51:23] sbiribizio: Check your /var/log/puppet.log [14:52:10] Found the problem [14:52:12] Thanx [16:43:21] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL 20.00% of data above the critical threshold [0.0] [16:43:39] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL 20.00% of data above the critical threshold [0.0] [16:43:59] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL 20.00% of data above the critical threshold [0.0] [16:44:41] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL 30.00% of data above the critical threshold [0.0] [16:44:47] PROBLEM - Puppet failure on tools-web-static-01 is CRITICAL 20.00% of data above the critical threshold [0.0] [16:45:07] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 22.22% of data above the critical threshold [0.0] [16:45:46] PROBLEM - Puppet failure on tools-shadow is CRITICAL 20.00% of data above the critical threshold [0.0] [16:45:56] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL 40.00% of data above the critical threshold [0.0] [16:46:00] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL 30.00% of data above the critical threshold [0.0] [16:46:08] PROBLEM - Puppet failure on tools-mail is CRITICAL 33.33% of data above the critical threshold [0.0] [16:46:40] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 40.00% of data above the critical threshold [0.0] [16:46:50] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL 40.00% of data above the critical threshold [0.0] [16:47:18] PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL 55.56% of data above the critical threshold [0.0] [16:47:38] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 55.56% of data above the critical threshold [0.0] [16:48:16] PROBLEM - Puppet failure on tools-submit is CRITICAL 66.67% of data above the critical threshold [0.0] [16:48:28] PROBLEM - Puppet failure on tools-webproxy-01 is CRITICAL 30.00% of data above the critical threshold [0.0] [16:48:44] !log shinken eh, which shinken process should I kill to silence shinken-wm_ ? did /etc/init.d/shinken stop now [16:48:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL, Master [16:49:05] that probably also stops email, but that's not the worst result [16:53:21] !log shinken so it's sudo /etc/init.d/ircecho stop. Restarting init.d/shinken. [16:53:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL, Master [18:10:34] Wikimedia Labs | Status: Openstack upgrade in progress;instance creation temporarily broken | Channel is logged: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs/ [18:18:05] andrewbogott, ^ [18:18:21] is that upgrade still going on? [18:18:30] yep [18:18:52] Huh, I thought it was 16:00-18:00 UTC [18:19:10] taken longer than expected? [18:19:38] sometimes the ticking of the clock doesn’t cause things to magically start working [18:20:51] heh [18:20:54] k [18:32:57] !log shinken alerts have subsided, restarting ircecho [18:33:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL, Master [18:33:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK Less than 1.00% above the threshold [0.0] [18:39:31] I have .bigbortherrc file, but bigbrother doesn't reinvoke job if it dies [18:41:45] eranroz: which project? [18:41:53] eranbot [18:42:29] (I just invoked the job myself) [18:43:03] eranroz: I think you need to remove the empty lines (see bigbrother.log) [18:44:35] DONE. thanks [18:59:43] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL 20.00% of data above the critical threshold [0.0] [18:59:51] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL 30.00% of data above the critical threshold [0.0] [19:02:30] PROBLEM - Puppet failure on tools-mailrelay-02 is CRITICAL 30.00% of data above the critical threshold [0.0] [19:02:33] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1621807 (10yuvipanda) After reading through the source for a bit I'm convinced that this is just because I was using the insecure port. I'll verify shortly. [19:03:26] PROBLEM - Puppet failure on tools-exec-1402 is CRITICAL 40.00% of data above the critical threshold [0.0] [19:03:40] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1405 is CRITICAL 20.00% of data above the critical threshold [0.0] [19:03:52] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 60.00% of data above the critical threshold [0.0] [19:04:23] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL 50.00% of data above the critical threshold [0.0] [19:05:01] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL 60.00% of data above the critical threshold [0.0] [19:06:05] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL 66.67% of data above the critical threshold [0.0] [19:06:17] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL 20.00% of data above the critical threshold [0.0] [19:07:11] andrewbogott: I think nodepool broke the labs metadata interface among others -:-((( [19:07:27] hashar: I don’t think you broke it, I think I broke it [19:07:33] PROBLEM - Puppet failure on tools-exec-1401 is CRITICAL 66.67% of data above the critical threshold [0.0] [19:07:50] attempting to create an instance via horizon yield an error saying there is no host left available and http://169.254.169.254/latest/ yields a 500 :D [19:07:51] ah [19:07:56] sorry for your lost! [19:08:09] PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL 55.56% of data above the critical threshold [0.0] [19:09:11] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 55.56% of data above the critical threshold [0.0] [19:09:15] PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 60.00% of data above the critical threshold [0.0] [19:09:37] PROBLEM - Puppet failure on tools-services-01 is CRITICAL 50.00% of data above the critical threshold [0.0] [19:09:49] PROBLEM - Puppet failure on tools-checker-02 is CRITICAL 22.22% of data above the critical threshold [0.0] [19:09:50] PROBLEM - Puppet failure on tools-services-02 is CRITICAL 40.00% of data above the critical threshold [0.0] [19:10:23] !log shinken disabling irc again [19:10:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL, Master [19:26:30] 6Labs, 7Shinken: Newly created instance is in ERROR state - https://phabricator.wikimedia.org/T111988#1621894 (10scfc) 3NEW [19:31:37] Hi, I was trying to set a job on labs and I'm getting: "perl: warning: Setting locale failed. [19:31:37] perl: warning: Please check that your locale settings:.....are supported and installed on your system." [19:31:45] 6Labs, 7Shinken: Newly created instance is in ERROR state - https://phabricator.wikimedia.org/T111988#1621949 (10yuvipanda) Openstack upgrade in progress, see IRC :) [19:32:29] Has that got something to do with the OpenStack upgrade? [19:33:28] Niharika: nope. [19:33:45] Unfortunately I've no idea about perl. Maybe coren can help [19:34:03] Niharika: what are you doing exactly? [19:34:53] valhallasw`cloud: Running a jsub .... command that I've run tens of times before, but I'm doing it from a different laptop. I don't think that should make a difference. [19:35:01] Let me make a pastebin. [19:35:17] it does [19:35:24] if your laptop locale is different [19:35:28] Oh? [19:35:47] But I SSHed into a remote machine. Why is it bothering about what my locale is? [19:35:58] because that's how SSH works [19:36:07] you get your local locale on the remote machine [19:36:17] what's the output of locale ? [19:36:28] valhallasw`cloud: Where can I set the locale. A couple of things are not set. Like LANGUAGE AND LC_ALL. [19:36:37] what's LANG? [19:36:51] locale: Cannot set LC_CTYPE to default locale: No such file or directory [19:36:52] locale: Cannot set LC_ALL to default locale: No such file or directory [19:37:12] LANG=en_US.UTF-8 [19:37:19] LANGUAGE is empty. [19:38:00] echo $LC_ALL ? [19:38:18] Blank. [19:38:58] huh. echo $LANG is en_US.UTF-8? [19:39:07] https://www.irccloud.com/pastebin/RIEb2SIl/ [19:39:27] yeah, that LC_CTYPE is odd [19:39:33] that should be en_US.UTF-8 [19:39:47] Okay. valhallasw`cloud - where can I fix this? [19:40:13] Niharika: wherever $LC_CTYPE is set. That's either on your local machine, or in your .bashrc or .profile [19:40:32] Okay. Checking. [19:46:30] valhallasw`cloud: Fixed. Thanks for the help! [19:46:41] yw! [19:55:20] 6Labs, 3Labs-sprint-112, 3ToolLabs-Goals-Q4, 3labs-sprint-113: Fix documentation & puppetization for labs NFS - https://phabricator.wikimedia.org/T88723#1622194 (10coren) There is WIP documentation at https://wikitech.wikimedia.org/wiki/Labs_NFS and, as far as I know, T102478 covered the puppetization. No... [20:04:42] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622254 (10Etune) Insecure port is likely reason for this. [20:07:34] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622273 (10Etune) You are setting " --insecure-bind-address=<%= @master_ip %> ". unless master_ip is 127.0.0.1, that's probably not doing what you want. [20:25:37] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Scaling: curl http://169.254.169.254/latest/meta-data/public-keys/ is unavailable - https://phabricator.wikimedia.org/T112001#1622352 (10hashar) 3NEW [20:26:10] wow... I just found references to tools.wikimedia.de in my code [20:29:19] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622390 (10yuvipanda) @Etune yup, that was the issue. Now at `F0909 20:26:59.469630 29907 helpers.go:71] Error in configuration: more than one authentication method found for ; found [token basicAut... [20:33:19] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622410 (10yuvipanda) I had previously used basic auth, and had the same user account there. I wonder if that is 'cached' somewhere (etcd perhaps?) despite the restarts? Doesn't look obvious from the... [20:41:50] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1622433 (10yuvipanda) 3NEW [20:42:39] 6Labs, 10Tool-Labs: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1622433 (10yuvipanda) Should basically track production branch as closely as possible (faster git pulls maybe?) [20:47:00] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622462 (10Etune) You probably need to edit your .kube/config [20:47:14] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622463 (10yuvipanda) Ok, that theory is bunk now. Still trying to figure out why it thinks basica uth is still on. [20:48:01] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622464 (10yuvipanda) my local user's .kube/config? doesn't exist... [20:48:20] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622465 (10Etune) you probably have a users[i].user.password and a users[i].user.token in that file. This is client side error, not apiserver side. [20:48:26] Coren, ping [20:49:20] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622466 (10yuvipanda) Am using: ```kubectl --insecure-skip-tls-verify -s https://tools-k8s-master-01.tools.eqiad.wmflabs:6443 --namespace='lolrrit-wm' --token=='' get pods``` [20:49:38] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622467 (10yuvipanda) (and had tried with --username and --user as well) [20:54:45] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622478 (10Etune) Are you seeing that fatal log message in the apiserver log file? [21:11:25] 6Labs, 10Tool-Labs: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1622588 (10yuvipanda) Nope in the client, when I try out kubectl. [21:12:57] 6Labs, 3Labs-sprint-112, 3ToolLabs-Goals-Q4, 3labs-sprint-113: Fix documentation & puppetization for labs NFS - https://phabricator.wikimedia.org/T88723#1622591 (10yuvipanda) Are you sure the shelves are connected to both hosts? I thought they weren't. [23:53:18] Coren, ping [23:57:12] 10Wikibugs: wikibugs - throttle output, don't get kicked for flooding - https://phabricator.wikimedia.org/T112032#1623218 (10Dzahn) 3NEW