[00:49:11] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Framawiki was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=686914 edit summary: 
[07:48:03] <wikibugs>	 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2407488 (10Niharika)
[09:42:28] <wikibugs>	 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2407653 (10yuvipanda)
[09:42:30] <wikibugs>	 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2407650 (10yuvipanda) 05Open>03Resolved a:03yuvipanda I've created it - it is mw-extension-ids, and added Niharika as admin. Feel free to add others.
[09:44:51] <wikibugs>	 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2407658 (10Niharika) Thanks!
[09:51:12] <zhuyifei1999_>	 andrewbogott: when you get on, can you check why this happened? https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Encoding03.video.eqiad.wmflabs&diff=686324&oldid=386634
[09:53:02] <andrewbogott>	 zhuyifei1999_: probably that virt node ran out of RAM.  We have more hardware getting installed next week.
[09:53:08] <andrewbogott>	 Were you able to restart the VM?
[09:53:29] <zhuyifei1999_>	 no, I'm not projectadmin
[09:54:24] <zhuyifei1999_>	 I only noticed it a few minutes ago when I failed to ssh in "no route to host"
[09:54:30] <andrewbogott>	 zhuyifei1999_: ok, let me move that to a different host and then I'll start it
[09:54:37] <zhuyifei1999_>	 k
[09:58:51] <yuvipanda>	 andrewbogott I've spawned 2 nodes now, I' want to spawn 4 more over the next few days
[10:01:35] <wikibugs>	 06Labs: confirm that new base labs base image is adequate for kubernetes &c. - https://phabricator.wikimedia.org/T134944#2407684 (10yuvipanda) a:05yuvipanda>03Andrew I can confirm it isn't enough - memory cgroups aren't enabled without a restart, and that's reasonably annoying.
[10:05:19] <andrewbogott>	 zhuyifei1999_: I didn't realize how gigantic that instance was… this is going to take probably 30-40 minutes.
[10:05:32] <andrewbogott>	 I need to step away but I have the migrate running automatically; in theory it should start up without me.
[10:05:38] <andrewbogott>	 I will check back after lunch to make sure it succeeded.
[10:05:43] <zhuyifei1999_>	 how gigantic?
[10:05:52] <andrewbogott>	 153G
[10:05:57] <zhuyifei1999_>	 omg
[10:06:16] <zhuyifei1999_>	 it's supposed to delete all the files after a task it done
[10:06:24] <Niharika>	 andrewbogott: I'm looking at instance commtech-1 on commtech.eqiad.wmflabs project and it's gone into limbo. Can't get it to reboot either.  
[10:06:40] <andrewbogott>	 well, it may not be storing that much right now, could be that it grew to that size sometime in the past
[10:06:45] <andrewbogott>	 Niharika: ok, one moment please
[10:06:47] <zhuyifei1999_>	 I haven't seen 100G+ in use at any time
[10:06:54] * andrewbogott has 10 minutes of battery life left
[10:07:16] <Niharika>	 andrewbogott: No rush, look into it whenever you get time. :)
[10:07:57] <zhuyifei1999_>	 (so apparently deleting stuffs doesn't prevent instance size from growing too much?)
[10:08:48] <andrewbogott>	 zhuyifei1999_: instance size == the biggest it has ever been
[10:08:52] <andrewbogott>	 so deleting things ongoingly is good
[10:08:59] <andrewbogott>	 but letting things grow and then cleaning up later doesn't help much
[10:09:09] <andrewbogott>	 unless you delete/recreate your instance periodically
[10:09:51] <andrewbogott>	 Niharika: 'gone into limbo'?
[10:10:04] <zhuyifei1999_>	 without puppet deleting and recreating is like a pain :(
[10:10:05] <Niharika>	 andrewbogott: Can't ssh in. Can't reboot. 
[10:10:10] <Niharika>	 Dunno what's up with it. 
[10:10:23] * Niharika has no more powers
[10:10:23] <andrewbogott>	 I can log in, looks fine to me
[10:10:45] <Niharika>	 andrewbogott: You can ssh in? Can you reboot it for me?
[10:10:52] <Niharika>	 It says "Failed to reboot"
[10:10:58] <andrewbogott>	 let me make sure that puppet is happy first...
[10:11:40] <andrewbogott>	 Niharika: rebooting
[10:11:56] <andrewbogott>	 !log commtech rebooting commtech-1 as per niharika's request
[10:12:00] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL, dummy
[10:12:58] <andrewbogott>	 Niharika: 
[10:13:02] <andrewbogott>	 root@commtech-1:~# uptime
[10:13:03] <andrewbogott>	  10:12:52 up 0 min,  1 user,  load average: 1.03, 0.23, 0.08
[10:13:06] <Niharika>	 Dummy? I thought it called you master. :) 
[10:13:11] <Niharika>	 andrewbogott: Thank you very much. 
[10:13:19] <andrewbogott>	 hope that helps — gotta go!
[10:13:32] <Niharika>	 Perfect! That works. 
[10:24:53] <wikibugs>	 06Labs, 10MediaWiki-Vagrant, 10MediaWiki-extensions-Newsletter, 13Patch-For-Review: "vagrant roles" commands fail with some encoding (LC_ALL) settings with "invalid byte sequence in US-ASCII" - https://phabricator.wikimedia.org/T131460#2407743 (10Tgr)
[10:51:53] <tom29739>	 !log tools.piagetbot Fixed the bot being creepy and sending random messages to channels.
[10:53:28] <tom29739>	 OK... is it ignoring me?
[11:13:38] <andrewbogott>	 labs-morebots: still there?
[11:13:39] <labs-morebots>	 I am a logbot running on tools-exec-1208.
[11:13:39] <labs-morebots>	 Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log.
[11:13:39] <labs-morebots>	 To log a message, type !log <msg>.
[11:13:47] <andrewbogott>	 tom29739: no idea :(
[11:14:18] <tom29739>	 It's logged before from me.
[11:15:04] <zhuyifei1999_>	 !log test-not-a-project
[11:15:04] <labs-morebots>	 Message missing. Nothing logged.
[11:15:10] <andrewbogott>	 zhuyifei1999_: your instance is migrated and back up
[11:15:14] <andrewbogott>	 sorry for the interruption
[11:15:22] <zhuyifei1999_>	 thx
[11:15:36] <zhuyifei1999_>	 ^ morebots is responding
[11:17:10] <zhuyifei1999_>	 andrewbogott: yep the instance seems good now
[11:38:24] <tom29739>	 !log tools.piagetbot Fixed the bot being creepy and sending random messages to channels.
[11:38:28] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.piagetbot/SAL, Master
[11:38:29] <tom29739>	 Oh well.
[11:39:29] <tom29739>	 It logged the first message.
[11:39:39] <tom29739>	 It just didn't say it had.
[12:06:19] <wikibugs>	 06Labs, 10MediaWiki-extensions-Newsletter: newsletter-test.wmflabs.org is broken - https://phabricator.wikimedia.org/T138686#2407308 (10Qgil) Apparently we break it at each hackathon.  :)
[12:06:25] <Niharika>	 yuvipanda: I created a new instance on my project but there's no "Instance ID" in the table. How do I ssh in? (I am sorry, I am sure I have done this before but I keep forgetting)
[12:06:55] <yuvipanda>	 hey
[12:07:01] <yuvipanda>	 Niharika what is the name of the instance?
[12:07:18] <Niharika>	 yuvipanda: ids on mw-extension-ids project.
[12:08:33] <yuvipanda>	 niharika https://wikitech.wikimedia.org/wiki/Help:Access might help. You also usually just ssh to '$instancename.$project.eqiad.wmflabs'
[12:08:44] <yuvipanda>	 so ids.mw-extension-ids.eqiad.wmflabs
[12:09:25] <Niharika>	 yuvipanda: I can't, it says key mismatch, permission denied. While I can login to another instance on a different project. 
[12:09:34] <Niharika>	 yuvipanda: Keys are universal across projects, correct?
[12:09:49] <yuvipanda>	 yeah, they should be
[12:10:05] <yuvipanda>	 let me look
[12:10:30] <Niharika>	 Yeah, it won't let me get in to that one. 
[12:10:32] <yuvipanda>	 Niharika can you click 'get console output' in the wikitech page and paste what's at the bottom?
[12:11:37] <Niharika>	 yuvipanda: Some ssh keys, safe to paste here?
[12:12:15] <yuvipanda>	 Niharika ah, hmm, paste without those?
[12:12:20] <yuvipanda>	 it should be but not srure
[12:12:28] <Niharika>	 yuvipanda: Pasted in PM. 
[12:12:34] <yuvipanda>	 ah
[12:12:43] <yuvipanda>	 niharika can you paste the other messages, that *aren't* the key?
[12:13:23] <Niharika>	 yuvipanda: Yep. 
[12:14:05] <Niharika>	 https://www.irccloud.com/pastebin/xy0nhFag/
[12:14:08] <Niharika>	 yuvipanda: ^
[12:14:38] <yuvipanda>	 niharika can you leave this one alone, create a bug, and create a new instance?
[12:15:15] <Niharika>	 yuvipanda: Sure. Another thing that might interest you - when I first created this instance, it went into ERROR state and I had to delete and re-spawn. 
[12:15:44] <yuvipanda>	 andrewbogott ^
[12:15:51] <yuvipanda>	 niharika yeah, we've been having stability issues of sorts...
[12:27:33] <tom29739>	 yuvipanda, Niharika, I was having that problem with my instance.
[12:29:05] <tom29739>	 I created a bug if you're interested in having a look: https://phabricator.wikimedia.org/T138438
[12:47:57] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#1874462 (10zhuyifei1999) Ubuntu Xenial 16.04 LTS seems to have `openjdk-8-jre`. Perhaps we can have exec nodes in xenial for this?
[12:51:10] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2407907 (10yuvipanda) Wikimedia as a whole has been moving to Debian for a while, so I do not think we'll support Ubuntu 16.04 ever.
[13:03:26] <wikibugs>	 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2407913 (10Peachey88)
[13:13:26] <wikibugs>	 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2407935 (10Shoichi) Thank you !
[14:31:59] <wikibugs>	 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2408077 (10Shoichi) >>! In T138679#2407650, @yuvipanda wrote: > I've created it - it is mw-extension-ids, and added Niharika as admin. Feel free to add others.  Excuse me, I have a question.  I guess  "mw-extension-ids" is a m...
[15:50:10] <wikibugs>	 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2408207 (10Niharika) Hi Shoichi, let's collaborate on this on irc or messenger. I can explain more. :)
[19:52:47] <Luke081515>	 enterprisey: heh, joining less channels that before? :D to your comment before: I read that everytiem too
[19:53:11] <enterprisey>	 Luke081515: yeah - did you spot that via whois, or what?
[19:53:40] <enterprisey>	 also, yeah, it's good to know someone else shares my Zuul pain
[19:53:49] <Luke081515>	 enterprisey: no, I just wanted to comment your comment, and saw you joining other channels, but not -operations
[19:53:52] <Luke081515>	 :D
[19:54:14] <Luke081515>	 enterprisey: heh, I read that always, since I'm from germany :D
[20:44:56] <wikibugs>	 06Labs: Request ids-extension Labs project - https://phabricator.wikimedia.org/T138679#2408572 (10Shoichi) >>! In T138679#2408207, @Niharika wrote: > Hi Shoichi, let's collaborate on this on irc or messenger. I can explain more. :)   Hi Niharika, so what's my next step to  use or set the "mw-extension-ids wiki" ?