[00:37:07] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-10 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:24:32] <shinken-wm>	 PROBLEM - Puppet staleness on tools-cron-01 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [43200.0]
[01:30:28] <shinken-wm>	 PROBLEM - Puppet staleness on tools-grid-master is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[01:33:47] <wikibugs>	 10Labs-Kubernetes, 10grrrit-wm: grrrit-wm update/deployment failing - https://phabricator.wikimedia.org/T133189#2307640 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Since it works now.
[01:34:39] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Decide on upgrade policy for Kubernetes - https://phabricator.wikimedia.org/T133598#2307643 (10yuvipanda) Since there are no objections, I'm going to setup a recurring window every two weeks for the k8s upgrade, preferably on Tuesdays.
[01:40:47] <shinken-wm>	 PROBLEM - Puppet staleness on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0]
[06:13:34] <wikibugs>	 06Labs, 10DBA, 10Horizon: TGR unable to login on Horizon - https://phabricator.wikimedia.org/T131630#2307795 (10jcrespo) a:05jcrespo>03None > "Chris suggests that something should be done to the 'users' table on silver"  What should be done? I have 0 context here.  Please do not assign it to me, apply th...
[11:05:20] <Mjbmr>	 any admin around?
[11:50:18] <Amir1>	 !log wikilabels manually rebooting wikilabels-staging-01.wikilabels.eqiad.wmflabs
[11:50:23] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master
[12:24:43] <Luke081515>	 andrewbogott: are you currently here?
[12:44:58] <Amir1>	 !log deploying e42885d to wikilabels
[12:44:58] <labs-morebots>	 deploying is not a valid project.
[12:51:04] <Amir1>	 !log wikilabels deploying e42885d to wikilabels
[12:51:09] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master
[12:53:25] <wikibugs>	 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2308672 (10jcrespo)
[13:07:43] <wikibugs>	 06Labs: Investigate labnet1002 kernel panic - https://phabricator.wikimedia.org/T135322#2308774 (10MoritzMuehlenhoff) 05Open>03Resolved
[13:12:27] <chasemp>	 !log tools reboot tools-exec-1220 stuck in state of unresponsivenss
[13:12:34] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[13:13:27] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[13:14:02] <wikibugs>	 06Labs, 10Tool-Labs: Figure out a way to support java 1.8 on tool labs (Merl's bot) - https://phabricator.wikimedia.org/T121279#2308803 (10BBlack) @Merl - Note: T105794#2294355 - deadlines for HTTPS are coming soon.  MerlBot continues to be on the shortlist of bots making the highest numbers of insecure reques...
[13:15:55] <Luke081515>	 chasemp: Currently not busy?
[13:16:08] <chasemp>	 currently busy, what's up?
[13:17:02] <Luke081515>	 chasemp: I want to request one floating IP for my project. I'm currently trying to setup an alternative for the actual IRC software for IRC-RC, as it is planned, and want to connect from outside too, to make tests
[13:18:06] <chasemp>	 I don't think that will be an issue but can you make a task so we have some history for the allocation? and @ me and andrew please
[13:18:36] <Luke081515>	 ok
[13:19:31] <wikibugs>	 06Labs: Request of one floating IP for project 'rcm' - https://phabricator.wikimedia.org/T135730#2308832 (10Luke081515)
[13:19:35] <Luke081515>	 chasemp: ^
[13:28:00] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[13:29:33] <chasemp>	 Luke081515: okidoke I'll discuss it w/ andrew when he's around
[13:31:36] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[13:33:38] <Luke081515>	 chasemp: ok, thanks
[13:45:32] <wikibugs>	 06Labs, 10Labs-Sprint-115, 10Tool-Labs, 10labs-sprint-116, and 2 others: Attribute cache issue with NFS on Trusty - https://phabricator.wikimedia.org/T106170#2308972 (10zhuyifei1999)
[13:54:45] <doctaxon>	 chasemp: what is the difference between -l release=trusty and -l release=precise ?
[13:55:52] <chasemp>	 doctaxon: one specifies you want your job run on a precise host, the the other a trusty host
[13:56:33] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0]
[13:57:06] <doctaxon>	 chasemp: yes, I know. But what are the differences between precise and trusty ?
[13:58:09] <chasemp>	 doctaxon: that question doesn't have an answer you want afaik, they are different releases of the same distro so packages, kernel, etc
[13:58:21] <chasemp>	 you can ask if a package is a different version and I could look that up 
[14:00:41] <doctaxon>	 ah okay
[14:03:26] <wikibugs>	 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2309076 (10Joe)
[14:03:34] <wikibugs>	 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2307710 (10Joe) p:05Triage>03Normal
[14:04:03] <wikibugs>	 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2307710 (10Joe) p:05Normal>03Low
[14:04:40] <wikibugs>	 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2307710 (10Joe) a:03Joe
[14:10:05] <wikibugs>	 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2309108 (10hashar) I have kept this one open in case #traffic want to investigate whether it can be a problem in production. For beta, the resta...
[14:36:14] <shinken-wm>	 PROBLEM - SSH on tools-webgrid-lighttpd-1408 is CRITICAL: Server answer
[14:45:26] <wikibugs>	 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2309220 (10Maathavan)
[14:53:29] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:54:34] <wikibugs>	 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2309281 (10yuvipanda) p:05Triage>03High Thank you for reporting this! Am looking into this right now.
[14:54:49] <YuviPanda>	 chasemp: worker nodes still have stale /public/dumps
[14:55:14] <chasemp>	 ah I grepped out exec and web that's why
[14:55:20] <chasemp>	 I have to hop on a call but
[14:56:23] <chasemp>	 "/bin/fuser -k /public/dumps; /bin/umount -f /public/dumps; /bin/mount -a; timeout 5s file /public/dumps/incr; echo $"
[14:56:26] <chasemp>	 YuviPanda: can you run that^
[14:57:49] <YuviPanda>	 yeah doing
[14:58:00] <chasemp>	 "/bin/fuser -k /public/dumps; /bin/umount -f /public/dumps; /bin/mount -a; timeout 5s test -e /public/dumps/incr; echo $"
[14:58:00] <chasemp>	 file => test -e
[14:58:07] <chasemp>	 file exits 0 even if there is nothing there :)
[14:58:15] <chasemp>	 not a functional change tho for remount itself
[14:59:05] <Luke081515>	 chasemp: do you know, at which time we get the new 16.04 ubuntu image? is that planned?
[14:59:21] <YuviPanda>	 Luke081515: nope, not planned afaik. just debian from now
[14:59:44] <Leah>	 You sure you want "echo $"?
[14:59:51] <Luke081515>	 YuviPanda: but for example vagrant won't run at debian
[14:59:57] <Luke081515>	 if you take a look at the docu
[15:00:31] <YuviPanda>	 Luke081515: mediawiki vagrant? it will at some point, since mediawiki in production is also moving to debian
[15:00:36] <chasemp>	 ah it cut off 'echo $?' Leah :)
[15:00:41] <Luke081515>	 hm
[15:00:44] <Leah>	 :-)
[15:01:14] <bd808>	 "mediawiki in production is also moving to debian" -- someday
[15:01:33] <YuviPanda>	 bd808: soon. _joe_ is working on it now I think, since he has new appservers and they're debian
[15:01:47] <YuviPanda>	 chasemp: hmm, an 'ls /public' still hangs
[15:01:49] <bd808>	 oh cool. I had not heard anything about it
[15:02:14] <bd808>	 I guess that means there is a big fixup project ahead for mw-vagrant
[15:02:33] <bd808>	 I know there are a lot of upstart jobs in there
[15:08:35] <YuviPanda>	 chasemp: I think they're all fucked now.
[15:08:42] <chasemp>	 all?
[15:08:45] <chasemp>	 I'm on a call atm
[15:08:46] <chasemp>	 what is all
[15:08:52] <YuviPanda>	 chasemp: kk I"ll investigate. all as in all worker nodes
[15:09:16] <YuviPanda>	 chasemp: no emergency etc. keep at call
[15:10:37] <wikibugs>	 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2309373 (10yuvipanda) @Maathavan does this work for you now?
[15:18:23] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Prevent breaking puppet on the dnsrecursor host when novaadmin isn't in a project - https://phabricator.wikimedia.org/T133946#2309403 (10Krenair) 05Open>03declined We decided against this - puppet breaking is fine and better than silently failing.
[15:25:52] <chasemp>	 YuviPanda: back now, what's the deal?
[15:26:09] <YuviPanda>	 chasemp: tools-worker-1001
[15:26:26] <YuviPanda>	 anything that touches dumps gets stuck
[15:26:37] <YuviPanda>	 (am also eating with one hand atm)
[15:28:20] <chasemp>	 uh yeah wth
[15:29:26] <chasemp>	 YuviPanda: may have to reboot?
[15:29:33] <chasemp>	 I don't get the current state at all
[15:29:49] <YuviPanda>	 chasemp: yeah ok. i depooled anyway
[15:30:00] <YuviPanda>	 chasemp: should we switch it to soft before?
[15:30:08] <YuviPanda>	 or doesn't matter...
[15:30:21] <chasemp>	 I'm coming in w/ a bit of refactor and that next across all for scratch  and dumps I hope
[15:30:32] <chasemp>	 but I'm doing a bit of testing on what the outcome is for changing nfs options on teh fly
[15:30:37] <chasemp>	 i.e. remount etc
[15:30:47] <chasemp>	 there is a race condition with it, but I'm unsure if we care
[15:30:52] <chasemp>	 long story :)
[15:31:29] <chasemp>	 also I have a different option set, you have timeo=10
[15:31:39] <chasemp>	 did you know that's 1/10th of a second?
[15:31:45] <YuviPanda>	 hahah really?
[15:31:46] <chasemp>	 so that means it times out now at .1 second
[15:31:47] <YuviPanda>	 no
[15:31:48] <chasemp>	 yes
[15:31:55] <YuviPanda>	 funny shit
[15:32:05] <YuviPanda>	 chasemp: can you reboot btw?
[15:32:08] <chasemp>	 so I'm thinking timeout=100,retrans=3
[15:32:09] <chasemp>	 am doing so
[15:32:45] <chasemp>	 I even got it to unmount
[15:32:50] <chasemp>	 but remounting for me or puppet was hanging
[15:32:53] <chasemp>	 so it' ssome kind of bug
[15:32:56] <chasemp>	 as that's not usual
[15:33:39] <YuviPanda>	 fun
[15:34:42] <chasemp>	 it's back and fine now
[15:35:04] <chasemp>	 but jessie has some unique issue here
[15:35:06] <YuviPanda>	 yeah and it's pooled itself too
[15:35:12] <YuviPanda>	 (as it is designed to)
[15:36:41] <chasemp>	 so I've been asking a lot about storage for candidates YuviPanda and the only one that really understood our overload issue w/ every client being able to do unlimited io and killing the cluster has been T
[15:36:48] <chasemp>	 and he basiclly said, yes it was a thing but they never solved it
[15:36:57] <chasemp>	 even ceph folks are kind of bewildered
[15:37:10] <YuviPanda>	 heh
[15:37:17] <chasemp>	 handing out direct storage to customers in this way seems very usual
[15:37:24] <YuviPanda>	 chasemp: btw, tools-worker-1012 has a different type of nfs issue now
[15:37:30] <YuviPanda>	 chasemp: /public/dumps won't mount
[15:37:36] <YuviPanda>	 even tho it isn't mounted
[15:37:46] <chasemp>	 does it hang?
[15:37:50] <chasemp>	 mount -a never returns
[15:37:51] <YuviPanda>	 chasemp: ya
[15:37:55] <YuviPanda>	 ya
[15:37:56] <chasemp>	 same issue then
[15:38:05] <chasemp>	 idk reboot is all I know for sure to fix it
[15:38:16] <chasemp>	 atm
[15:38:19] <YuviPanda>	 I guess this reinforces my idea that touching anything NFS is never ever simple :)
[15:38:27] <YuviPanda>	 ya rebooting
[15:39:14] <chasemp>	 so  this relats to mystery /mnt/nfs :)
[15:39:28] <chasemp>	 I'm looking at putting all nfs mounts under /mnt/nfs as I don't like them at top level like /home
[15:39:35] <YuviPanda>	 chasemp: looks like I have to reboot all the k8s workers now :(
[15:39:40] <chasemp>	 YuviPanda: hm ok
[15:39:47] <chasemp>	 I'll try to keep in mind to test on jessie too for things
[15:40:00] <YuviPanda>	 chasemp: or not, since a reboot itself hangs...?!
[15:40:05] <YuviPanda>	 at least a sudo reboot hangs
[15:40:11] <chasemp>	 huh
[15:41:02] <YuviPanda>	 I'm just going to reboot from horizon now
[15:41:06] <chasemp>	 ok man
[15:41:19] <YuviPanda>	 aaaa except I can't since I just wiped my phone again this morning.
[15:41:24] <chasemp>	  I wish I had a good answer here
[15:41:38] <grrrit-wm>	 (03PS1) 10BryanDavis: jsub: validate '-l release=...' arguments [labs/toollabs] - 10https://gerrit.wikimedia.org/r/289677 
[15:41:42] <YuviPanda>	 I'm going to reboot from nova
[15:41:47] <chasemp>	 heh ok
[15:42:17] <grrrit-wm>	 (03Abandoned) 10BryanDavis: jsub: Add support for qsub args used by tools.merlbot [labs/toollabs] - 10https://gerrit.wikimedia.org/r/288226 (https://phabricator.wikimedia.org/T135006) (owner: 10BryanDavis)
[15:43:18] <YuviPanda>	 !log tools rebooting all tools worker instances
[15:43:23] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[15:43:25] <grrrit-wm>	 (03CR) 10Rush: [C: 031] "not tested but looks sane to me, thanks bryan" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/289677 (owner: 10BryanDavis)
[15:43:31] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] jsub: validate '-l release=...' arguments [labs/toollabs] - 10https://gerrit.wikimedia.org/r/289677 (owner: 10BryanDavis)
[15:44:15] <shinken-wm>	 PROBLEM - SSH on tools-worker-1004 is CRITICAL: Connection refused
[15:46:18] <shinken-wm>	 PROBLEM - SSH on tools-worker-1009 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:16] <shinken-wm>	 RECOVERY - SSH on tools-worker-1004 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0)
[15:51:08] <shinken-wm>	 RECOVERY - SSH on tools-worker-1009 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0)
[16:02:03] <zhuyifei1999_>	 YuviPanda: that reminds me of https://phabricator.wikimedia.org/T130577, in that case sudo reboot itself also hangs, but it's not an nfs instance
[16:04:18] <zhuyifei1999_>	 (and sudo kill -9 on hanging processes does nothing)
[16:24:06] <wikibugs>	 06Labs: Request of one floating IP for project 'rcm' - https://phabricator.wikimedia.org/T135730#2309644 (10Andrew) 05Open>03Resolved a:03Andrew ok, done.
[16:43:52] <wikibugs>	 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2309716 (10yuvipanda) I have had to restart all the notebooks due to some NFS issues. Re-logging in should work now. Apologies for the issues!
[17:06:39] <bd808>	 !log tools.hatjitsu Stopped competing web jobs; started with `webservice nodejs start`
[17:06:44] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hatjitsu/SAL, Master
[18:23:06] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[19:28:46] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 031] "Needs a changelog bump for deployment tho." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/289677 (owner: 10BryanDavis)
[19:34:14] <grrrit-wm>	 (03PS3) 10BryanDavis: jsub: validate '-l release=...' arguments [labs/toollabs] - 10https://gerrit.wikimedia.org/r/289677 
[19:38:25] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] jsub: validate '-l release=...' arguments [labs/toollabs] - 10https://gerrit.wikimedia.org/r/289677 (owner: 10BryanDavis)
[19:38:36] <YuviPanda>	 bd808: \o/ will you do the deploy at some point?
[19:38:56] <grrrit-wm>	 (03Merged) 10jenkins-bot: jsub: validate '-l release=...' arguments [labs/toollabs] - 10https://gerrit.wikimedia.org/r/289677 (owner: 10BryanDavis)
[19:40:27] <bd808>	 YuviPanda: Yeah. Let me see if I can follow the instructions I wrote but didn't actually test :)
[19:44:30] <bd808>	 where do I get a list of all the bastions? Just from looking at the instance list?
[19:45:00] <YuviPanda>	 Bd808 yeah
[19:45:12] <YuviPanda>	 Easiest I think
[19:52:08] <bd808>	 YuviPanda: aptly error -- ERROR: unable to publish: unable to process packages: error linking file to /srv/packages/public/pool/main/t/tools-manifest/tools-manifest_0.6_all.deb: file already exists and is different
[19:57:17] <YuviPanda>	 bd808 ugh. Just RM that file for now
[19:57:21] <YuviPanda>	 That makes it work
[19:57:29] <YuviPanda>	 I need to fix that at some point
[19:57:37] <bd808>	 I have an idea of what causes it
[19:58:06] <YuviPanda>	 A mistake a I made early on where I tried to re-add the same deb version for two different builds
[19:58:06] <bd808>	 aptly doesn't seem to like getting 2 builds of a universal package
[19:58:21] <YuviPanda>	 It succeeded and then doesn't stop complaining since
[19:58:57] <YuviPanda>	 Really? If I RM it it just works usually
[19:59:56] <bd808>	 you rm target file?
[20:00:57] <YuviPanda>	 The file it complains about
[20:01:25] <YuviPanda>	 The old tools manifest deb
[20:01:34] <bd808>	 ok
[20:02:40] <bd808>	 I'm pretty sure the problem is that `find /srv/packages/pool -name tools-manifest_0.6_all.deb` returns 2 _all pcakges
[20:03:07] <bd808>	 I just caused the same thing for this new build :)
[20:03:12] <YuviPanda>	 Haha
[20:03:15] <YuviPanda>	 Fun
[20:03:39] <YuviPanda>	 Debs are from a different era imo
[20:03:57] <YuviPanda>	 For things like this that is
[20:05:04] <bd808>	 blarg
[20:05:25] <bd808>	 now I've got fresh problems. /me debugs
[20:05:40] <bd808>	 I rm'd a dup and made the index mad
[20:10:01] <bd808>	 All better I think
[20:14:47] <YuviPanda>	 \o/
[20:14:54] <YuviPanda>	 bd808: can you add it to the Aptly page on wikitech?
[20:29:22] <bd808>	 YuviPanda: https://wikitech.wikimedia.org/w/index.php?title=Aptly&diff=544645&oldid=501052
[20:30:02] <YuviPanda>	 bd808: <3 thanks
[20:31:49] <bd808>	 aargh. load on bastion-02 is 66
[20:32:43] <bd808>	 why are there 61 copies of sessionclean running?
[20:33:15] <bd808>	 I know this bug from somewhere ...
[20:34:16] <bd808>	 Ah T73645
[20:34:17] <stashbot>	 T73645: php5 session cleanup script can go nuts - https://phabricator.wikimedia.org/T73645
[20:34:33] <bd808>	 Which I fixed in mw-vagrant with https://gerrit.wikimedia.org/r/#/c/164877/
[21:07:21] <bd808>	 !log tools deployed jobutils 1.13 on bastions; now with '-l release=...' validation!
[21:07:27] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[22:35:02] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:13:58] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2311122 (10bd808)
[23:15:05] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2311140 (10bd808)
[23:15:20] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2311122 (10bd808)
[23:15:24] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion, 15User-bd808: Create application to manage Diffusion repositories for a Tool Labs project - https://phabricator.wikimedia.org/T133252#2226182 (10bd808)
[23:17:27] <CP678|Laptop>	 YuviPanda, so I just had a very strange experience when I logged into my cyberbot exec node on my cyberbot project using WinSCP
[23:18:07] <CP678|Laptop>	 It SSH'd into Bastion just fine, and then proceeded to continue to the exec node only to tell my the fingerprint on it didn't match the currently saved one.
[23:18:22] <CP678|Laptop>	 I aborted, naturally
[23:18:30] <CP678|Laptop>	 When I tried again, it worked just fine.
[23:19:05] <CP678|Laptop>	 Since it went into Bastion successfully, I'm assuming something there went wrong momentarily.
[23:19:23] <CP678|Laptop>	 I'm unable to replicate it, but I thought you should know