[00:02:28] Coren: Is jouncebot meant to read only the deployments calendar? In order for me to make it read another calendar, do I have to install it myself, or will creating a YAML file suffice? [00:03:16] bmansurov: I... have no idea. I've never played with that bot before. I expect the current maintainer could tell you, though. [00:03:51] Coren: ok thank you [00:04:23] YuviPanda|brb: yt? [00:04:27] I am :D [00:05:44] but I don’t know anything about jouncebot :) [00:05:44] YuviPanda: I saw your name in the list of commiters ;) [00:05:44] bmansurov: greg-g’s team maintains it, not sure who in particular. I’d suggest just reading through the code :) [00:05:44] I made a purely joke commit :D [00:05:45] YuviPanda: oh ok, thanks [00:05:57] greg-g: hey, can I ask you a question about jouncebot? Or do you know anyone who I could ask? [00:08:47] Coren: Can you add me to the 'tools.jouncebot' group? Or should I ask one of the maintainers of jouncebot? [00:14:51] bmansurov: ask an existing maintainer. [00:15:06] T13|away: ok thanks [00:16:25] @link [[meta:Requests for comment/Abandoned Labs tools]] for more info. [00:16:25] https://meta.wikimedia.org/wiki/Requests_for_comment/Abandoned_Labs_tools [00:18:02] thanks [00:55:57] 6Labs, 10Tool-Labs, 3Labs-Q4-Sprint-2, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Review and productionize webservice manifest monitor - https://phabricator.wikimedia.org/T95210#1205449 (10yuvipanda) I need to write the email, and that somehow feels like a lot more work than writing all the code. Also needs... [01:41:12] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Provide a status page (list) of all active proxy definitions - https://phabricator.wikimedia.org/T88216#1205522 (10yuvipanda) Note that we can also have a list of webservices that *should* be running by looking at service manifests :) [01:50:48] !ping [01:50:48] !pong [01:56:55] 10Tool-Labs, 3Labs-Q4-Sprint-1, 3ToolLabs-Goals-Q4: Add more Trusty exec nodes - https://phabricator.wikimedia.org/T94304#1205548 (10yuvipanda) 5Open>3Resolved Well, this was done. I'll open up another bug for the VMEM issue. [01:58:02] 10Tool-Labs, 3ToolLabs-Goals-Q4: Harmonize VMEM available on all exec hosts - https://phabricator.wikimedia.org/T95979#1205550 (10yuvipanda) 3NEW [01:58:11] 10Tool-Labs, 3Labs-Q4-Sprint-1, 3ToolLabs-Goals-Q4: Add more Trusty exec nodes - https://phabricator.wikimedia.org/T94304#1205558 (10yuvipanda) (T95979) [02:01:18] Deskana: I presume you’ve left the office [02:23:11] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Provide a status page (list) of all active proxy definitions - https://phabricator.wikimedia.org/T88216#1205565 (10yuvipanda) I've backported Jessie's lua-json to tools' local trusty repo and all is well. [02:38:08] 6Labs, 10Gerrit-Migration, 6Phabricator: Stabilize vcs-user owned files and directories - https://phabricator.wikimedia.org/T95982#1205584 (10Negative24) 3NEW a:3Negative24 [02:39:04] 6Labs, 10Gerrit-Migration, 6Phabricator: Stabilize vcs-user owned files and directories - https://phabricator.wikimedia.org/T95982#1205595 (10Negative24) [02:39:17] 6Labs, 10Gerrit-Migration, 6Phabricator: Stabilize vcs-user owned files and directories in Phab-02 - https://phabricator.wikimedia.org/T95982#1205597 (10Negative24) [03:14:41] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Get rid of portgranter - https://phabricator.wikimedia.org/T93046#1205666 (10yuvipanda) Alright, so uwsgi and nodejs tools now pick up from random port :) As for list.php, an accurate way to do it now would be to read service.manifest file in that tool's h... [03:36:45] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Get rid of portgranter - https://phabricator.wikimedia.org/T93046#1205701 (10yuvipanda) Bit the bullet and did it. list.php is going to be possibly out of date for a bit now. That's ok, I think. [03:48:18] 10Tool-Labs, 7Puppet: Develop and publish a gridengine provider for Puppet - https://phabricator.wikimedia.org/T95525#1205726 (10yuvipanda) p:5Low>3Lowest I personally still think that a proper solution for the gridengine puppetization problem is to not use gridengine :) [04:09:41] (03PS1) 10Yuvipanda: Temporarily remove 'web' link URLs from main page listing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/204013 (https://phabricator.wikimedia.org/T93046) [04:09:43] (03CR) 10jenkins-bot: [V: 04-1] Temporarily remove 'web' link URLs from main page listing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/204013 (https://phabricator.wikimedia.org/T93046) (owner: 10Yuvipanda) [04:09:57] (03PS2) 10Yuvipanda: Temporarily remove 'web' link URLs from main page listing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/204013 (https://phabricator.wikimedia.org/T93197) [04:09:59] (03CR) 10jenkins-bot: [V: 04-1] Temporarily remove 'web' link URLs from main page listing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/204013 (https://phabricator.wikimedia.org/T93197) (owner: 10Yuvipanda) [04:10:55] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Make list.php not rely on portgranter - https://phabricator.wikimedia.org/T93197#1205753 (10yuvipanda) Wait, it seems to open that file but I don't see it being used anywhere. Huh? [04:12:06] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Make list.php not rely on portgranter - https://phabricator.wikimedia.org/T93197#1205754 (10yuvipanda) Nevermind, I can't grep. [04:15:31] (03PS3) 10Yuvipanda: Do not depend on portgranter [labs/toollabs] - 10https://gerrit.wikimedia.org/r/204013 (https://phabricator.wikimedia.org/T93197) [06:02:18] Topic looks outdated. [06:39:40] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:04:41] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [07:42:51] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1205875 (10Thgoiter) 3 hours down yesterday [08:45:01] 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 3Continuous-Integration-Isolation: Support dedicating a specific virt node to a specific nova project - https://phabricator.wikimedia.org/T84989#1205967 (10hashar) I once filled a bug (T59833) to get the bastion instances to be on different hosts, th... [09:38:50] 10Tool-Labs: /etc/cron.daily/logrotate: gzip: stdin: file size changed while zipping - https://phabricator.wikimedia.org/T96007#1206058 (10valhallasw) 3NEW [10:55:41] Is it just me or is labs really slow? [10:58:08] T13|mobile: no noticeable slowness for me [11:38:54] 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 3Continuous-Integration-Isolation: Support dedicating a specific virt node to a specific nova project - https://phabricator.wikimedia.org/T84989#1206253 (10hashar) Related change by @andrew @chasemp [[ https://gerrit.wikimedia.org/r/#/c/203969/ | Ger... [12:24:45] PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:54:42] RECOVERY - Puppet failure on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:53] [13intuition] 15Krinkle pushed 2 new commits to 06master: 02https://github.com/Krinkle/intuition/compare/cad7062fb1fb...1a1d9a86f84b [13:46:53] 13intuition/06master 144f5ba14 15Timo Tijhof: dashboard: Update graph to cover 8 months instead of 4 weeks [13:46:54] 13intuition/06master 141a1d9a8 15Timo Tijhof: build: Add php 5.6 to Travis, include grunt-cli in package.json [13:52:58] andrewbogott_afk: Do you still have that list-of-precise suitable for use in salt? [13:59:05] !db [13:59:05] http://bots.wmflabs.org/~petrb/db/%23wikimedia-labs.htm [13:59:44] !database [14:04:15] [13intuition] 15Nikerabbit created 06json-i18n2 (+1 new commit): 02https://github.com/Krinkle/intuition/commit/83b5336d832c [14:04:16] 13intuition/06json-i18n2 1483b5336 15Niklas Laxström: Convert messages to json [14:22:18] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration: Diamond metrics for cpu.system suddenly up 100% after a reboot - https://phabricator.wikimedia.org/T95912#1206480 (10Krinkle) p:5Triage>3Unbreak! [14:22:38] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration: Diamond metrics for cpu.system suddenly up 100% after a reboot - https://phabricator.wikimedia.org/T95912#1206482 (10Krinkle) p:5Unbreak!>3Normal [14:24:50] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration: Diamond collected metrics about memory usage inaccurate until third reboot - https://phabricator.wikimedia.org/T91351#1206489 (10Krinkle) p:5Triage>3Low [14:39:27] Change on 12www.mediawiki.org a page Developer access was modified, changed by ANYKL link https://www.mediawiki.org/w/index.php?diff=1534589 edit summary: [+1840] [14:46:05] Change on 12www.mediawiki.org a page Developer access was modified, changed by ANYKL link https://www.mediawiki.org/w/index.php?diff=1534594 edit summary: [+670] [14:50:40] Change on 12www.mediawiki.org a page Developer access was modified, changed by MaxSem link https://www.mediawiki.org/w/index.php?diff=1534599 edit summary: [-2510] Reverted edits by [[Special:Contributions/ANYKL|ANYKL]] ([[User talk:ANYKL|talk]]) to last revision by [[User:Shirayuki|Shirayuki]] [15:20:33] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1206701 (10Krinkle) 5declined>3Open https://integration.wikimedia.org/ci/job/npm/2590/console ``` 00... [15:39:29] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1206779 (10hashar) On CI we do DNS requests for pubic DNS entry so we apparently need to remove the `ndo... [15:56:46] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1206842 (10Krinkle) 5Open>3Resolved [16:16:21] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Provide a status page (list) of all active proxy definitions - https://phabricator.wikimedia.org/T88216#1206928 (10yuvipanda) Should this be closed now or should it also provide a publicly viewable page? [16:26:04] What's up with the timescale tool maintained by Smith609? [16:26:42] It's operation on WP is leading to an error page. [16:27:18] hi [16:27:19] looking [16:27:30] See WP Template Talk:Ga [16:27:36] looks like it had been dead for quite a while [16:27:38] * YuviPanda starts it [16:27:49] I mean See WP Template Talk:Ma [16:30:33] The timescale tool is not an error page now, but its rendering is garbled. [16:30:51] How does it look to you? (I've JavaScript turned on) [16:30:52] Guest79897: yeah, try again now [16:31:14] I moved it to a newer version of PHP and that utterly broke it so moved it back [16:31:31] looks ok now [16:31:31] He he he. IT works. [16:42:52] 10Tool-Labs, 3Labs-Q4-Sprint-3, 5Patch-For-Review, 3ToolLabs-Goals-Q4: Get rid of portgranter - https://phabricator.wikimedia.org/T93046#1206976 (10yuvipanda) [16:43:51] 6Labs, 10Tool-Labs, 3Labs-Q4-Sprint-2, 3ToolLabs-Goals-Q4, 7Tracking: Replace bigbrother and ssh-cron-thingy with service manifests - https://phabricator.wikimedia.org/T90561#1206977 (10yuvipanda) So webservices are done now. Need to set tasks up for cron and worker services. [16:44:32] How do I get a bot cloak? [16:46:06] I'd like to be @wikimedia/bots/xTools [16:46:16] xTools: use the same request form as normal IRC cloaks [16:46:19] You are root identified by name .*@wikimedia/Technical-13 [16:46:19] @whoami [16:46:22] Okay [17:02:34] Hey MusikAnimal [17:02:43] howdy [17:03:29] 6Labs, 3Labs-Q4-Sprint-3: Labs: puppetize stripe_cache_size tweaks on labstores - https://phabricator.wikimedia.org/T96045#1207061 (10coren) 3NEW a:3coren [17:09:07] 6Labs, 6operations, 10ops-eqiad: labvirt100x boxes 'no carrier' on eth1 - https://phabricator.wikimedia.org/T95973#1207090 (10Cmjohnson) Added second eth cable ge-3/0/8 up up labvirt1001 eth0 ge-3/0/20 up up labvirt1002 eth0 ge-3/0/21 up up labvirt1003 eth0 ge-3/0/37... [17:13:40] Betacommand: you here? [17:16:55] Technical_13: Yeah [17:17:26] Can you come to #xTools? Have a db q for you [17:18:20] @link [[phab:T91320]] related [17:18:20] https://wikitech.wikimedia.org/wiki/phab:T91320 [17:18:21] YuviPanda: andrewbogott: gotta do a quick bank run to finally fix that stupid US card thing. BBIAB (~1.5h or so) [17:18:30] Coren: cool [17:18:46] YuviPanda: good luck! [17:19:01] andrewbogott: you meant that for Coren right? :) [17:19:13] um… yes. [17:19:18] :) [17:20:58] RECOVERY - Host tools-webproxy-jessie is UP: PING OK - Packet loss = 0%, RTA = 3.52 ms [17:21:07] lolwat [17:21:11] that host shouldn’t exist at all [17:23:06] 6Labs, 10hardware-requests, 6operations: Replace virt1000 with a newer warrantied server - https://phabricator.wikimedia.org/T90626#1207139 (10RobH) [17:24:00] 6Labs, 10hardware-requests, 6operations: Replace virt1000 with a newer warrantied server - https://phabricator.wikimedia.org/T90626#1207140 (10RobH) 5Open>3Resolved I've created the setup task T96048 for setting up system wmf3152 as labcontrol1001. I'm resolving this task, as hardware has been allocated. [17:24:04] 6Labs, 3ToolLabs-Goals-Q4: virt1000 SPOF - https://phabricator.wikimedia.org/T90625#1207152 (10RobH) [18:08:20] 10Tool-Labs, 3ToolLabs-Goals-Q4: Make proxylistener not need to keep open socket connections open - https://phabricator.wikimedia.org/T96059#1207356 (10yuvipanda) 3NEW [18:14:55] Did something just happen? [18:15:40] wm-bot2 wm-bot3 and wm-beta all just puked. [18:16:25] !ping [18:16:25] !pong [18:16:30] Technical_13: not for overall labs... [18:16:40] Okay. [18:27:47] 10Tool-Labs, 3Labs-Q4-Sprint-3, 3ToolLabs-Goals-Q4: Make proxylistener not need to keep open socket connections open - https://phabricator.wikimedia.org/T96059#1207419 (10yuvipanda) Am trying out gridengine epilog scripts to see if those would be enough. [18:39:10] 6Labs, 6operations: Investigate ways of getting off raid6 for labs store - https://phabricator.wikimedia.org/T96063#1207452 (10yuvipanda) 3NEW [18:47:12] 6Labs, 6operations: Investigate heavy NFS users and see if they can move IO to local storage - https://phabricator.wikimedia.org/T96065#1207477 (10yuvipanda) 3NEW [19:09:49] YuviPanda: these cron emails are basically useless the way they are :/ [19:10:19] Yes I put them out of my inbox [19:10:26] Actually no [19:10:31] I do that for root [19:10:42] Admin cron emails still make it [19:11:08] and that grid email is incomprehensible as well, but that might just be me [19:11:21] No [19:11:22] :p [19:12:31] hey, I just realized we have a way to send people cron-like emails from sge [19:12:36] by using and epilog script [19:12:38] -d [19:13:28] but! a broken epilog script puts the queue in an error state, so let me check that... [19:16:22] nothing in error state, though [19:19:57] is there an update on what/who is always filling up tools-redis? It gets a bit annoying that it's always out of memory and evicts keys very quickly [19:21:25] sitic: no, I don't think so. [19:22:42] sitic: I also don't think we really have the in-house knowledge on how to do that, which might explain why it takes a bit long [19:24:31] valhallasw`cloud: do you think you can take a look at that? I just haven’t had the time at all :( [19:26:03] yes it doesn't seem that redis is build for such audit usecases. Maybe two redis instances would be better (one for "cache" usecases, and one for more semi-persistent database usecases)? [19:26:13] yup [19:26:24] problem again is one of time, sadly [19:26:34] sitic: or redis-cluster [19:27:40] as always :-/ just wanted to give a ping that some users care about that redis server ;-) [19:28:42] :D totally [19:28:53] redis-cluster is also maybe the ‘right’ solution [19:30:12] grr, mosh doesn't support agent forwarding [19:30:35] yup :) [19:30:45] been there, felt that pain [19:32:08] where is the redis data to begin with... not in /var/redis [19:32:23] valhallasw`cloud: /srv [19:33:59] * Coren idly wonders if everyone who works in banking are morons or if they are obtuse on purpose. [19:34:20] a little bit from column A, a little bit from column B [19:37:37] virtualenv not installed on tools-redis [19:37:38] * valhallasw`cloud rages [19:38:14] 6Labs, 10Tool-Labs, 7Tracking: Make toollabs reliable enough (Tracking) - https://phabricator.wikimedia.org/T90534#1207579 (10yuvipanda) [19:38:20] valhallasw`cloud: :P [19:38:21] one does not simply not have virtualenv available [/jk] [19:38:46] valhallasw`cloud: also not all redis commands are available [19:38:51] I know [19:38:55] I copied the dump file instead [19:38:57] 6Labs, 6operations: Investigate ways of getting off raid6 for labs store - https://phabricator.wikimedia.org/T96063#1207581 (10coren) p:5Triage>3Low Raid 6 is a performance bottleneck but gives us 66% more effective storage than raid10 would in the current configuration. It doesn't mean that moving away f... [19:38:59] oh nice :) [19:39:04] there's a python tool to parse the dumps [19:39:07] oh nice [19:39:12] hence venv :P [19:46:54] YuviPanda: is that restricted task still private now that it's resolved? [19:48:44] 10Tool-Labs: Audit redis usage on toollabs - https://phabricator.wikimedia.org/T91979#1207615 (10valhallasw) Made a copy of the redis db to /srv/redis/tools-redis-valhallasw-dump.rdb. Now running rdb -c memory [1] to see if there's any clear memory hog. [1] ttps://github.com/sripathikrishnan/redis-rdb-tools [19:51:11] andrewbogott: Hm, did you ever get my question? "Do you still have a list of precise instances usable for a salt run handy"? [19:51:43] Coren: missed it, sorry. I can make a new list, hang on [19:52:34] the query takes ~5 minutes to run [19:53:28] Danke. [19:53:40] Coren: meanwhile… what syntax should I be using for my if ! defined line here? https://gerrit.wikimedia.org/r/#/c/204067/ [19:54:04] I'm going to do a salt run to remount all the NFS on the instances that it will be able to - fewer to reboot. [19:54:06] * Coren checks. [19:55:22] YuviPanda: 'ERR unknown command 'client'' when I try to do client list. Is that one of the disabled commands...? [19:55:45] because it's supposed to exist for redis 2.4+ and we're running 2.8 [19:56:19] andrewbogott: I think your problem is just the single colon between Ssh and userkey (as opposed to 2) [19:56:59] ah, that’s a good point — but I fixed that locally and the parser is still unhappy. Syntax error at '::userkey'; expected ')' [19:57:41] I don’t see any other examples in the code of checking the definition of a qualified::class [19:57:44] so maybe that just can’t be done :( [19:57:51] well, ok, not a class, a define [19:57:56] Hm. afaik, that should work but perhaps the ssh module hasn't been parsed yet? defined() is very order-dependent [19:58:34] YuviPanda: oh, right, that's in the disabled commands thing. Beh. [20:00:05] * andrewbogott devotes today’s 2 Minute Hate to puppet [20:00:24] 10Tool-Labs: Audit redis usage on toollabs - https://phabricator.wikimedia.org/T91979#1207634 (10valhallasw) Another idea is to use CLIENT LIST and INFO together to correlate increase in memory use with the client used, but the CLIENT command is disabled. [20:04:21] andrewbogott: It might be worthwhile to explicitly 'class Ssh' first? It would make certain the class is in the local context? [20:06:58] I’ll just rearrange things and avoid the question [20:10:24] coren: virt1000:/root/novastats/preciseinstances.txt [20:10:42] andrewbogott: Thankee [20:32:45] 6Labs, 6operations: Investigate heavy NFS users and see if they can move IO to local storage - https://phabricator.wikimedia.org/T96065#1207759 (10BBlack) I think we've gone around looking for these at various times in the past and always been able to cull a few of the worst offenders. I was thinking more alo... [21:17:54] Is there currently a known problem that prevents the creation of new instances? [21:21:41] physikerwelt: No, but are you sure you have the quota for the size you are trying to create? [21:23:15] andrewbogott: What irc bouncer software do you use? [21:24:18] 6Labs, 6operations: Investigate ways of getting off raid6 for labs store - https://phabricator.wikimedia.org/T96063#1207942 (10BBlack) As we've seen, RAID6 has serious performance issues under write heavy load, and isn't especially great at recovering from unclean shutdown (but we could address the latter to... [21:24:31] Negative24: I use dircproxy but I don’t love it. [21:24:54] * Coren uses znc and does love it. [21:24:56] Coren: I think so [21:25:09] andrewbogott: Alright. Coren, that's what I'll probably use [21:25:29] 'cause as you know I come and go quite often :) [21:25:50] and then I end up having to read logs [21:26:40] Coren: Is there additional quota information beside https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=displayquotas&projectname=math [21:29:00] My last attempt to create an instance lead to https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:I-00000af7.eqiad.wmflabs&action=edit&redlink=1 [21:29:26] andrewbogott: ^^ something you are aware of? [21:29:40] There is no information at all about what has happend [21:30:01] it would be bad if there was an instance that is now detached from project [21:30:32] physikerwelt: obviously i can’t tell what/where you tried to create from that link :) [21:31:16] andrewbogott: can you see if an instance I-00000af7 has been created? [21:31:41] physikerwelt: what name and project? [21:31:43] I was following this link to create an instance https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=create&project=math®ion=eqiad [21:31:47] math [21:34:04] physikerwelt: what flavor did you use? [21:34:17] That instnace creation failed, as evidenced by the big ERROR on the instance management page :) [21:35:25] andrewbogott: I saw the error... but it was not very specific. I used m1.medium with ubuntu14.04 security groups default [21:36:09] I tried the same thing yesterday and had a similar error [21:37:05] physikerwelt: yeah, something interesting is happening. Looking [21:37:19] Openstack has decided that ‘medium’ is too big for everything [21:39:53] andrewbogott: As long as no instances have been created this is no big deal. Otherwise there are now at least 3 instances that are around without any connection to a project [21:41:03] physikerwelt: try now? [21:41:43] great works now i-00000afa [21:42:04] my mistake — left some settings in place from a failed experiment yesterday [21:44:03] No problem. Maybe the instance state could be a little bit more specific. NOT CREATED or UNBORN would be a better description than ERROR which somehow implies to me that there is at least an instance [21:48:11] andrewbogott: Anyhow thank you. Have a nice day. [22:32:13] I love bouncers. I have detached four times now but I don't think I disconnected. [22:32:32] Negative24: Nice. [22:32:34] Coren: and yes. ZNC is amazing [22:33:45] * T13|mobile is running popcorn [22:35:20] I love bouncers, except when they kick me out of clubs :( [22:35:32] :D [22:36:09] https://en.wikipedia.org/wiki/Bouncer_%28cricket%29 [22:37:37] silent disco, when the DJ sets mode +m [22:43:21] 6Labs: labvirt1001 network config problem - https://phabricator.wikimedia.org/T96097#1208205 (10Andrew) 3NEW a:3Cmjohnson [22:46:27] 10Tool-Labs: Configure web services in such a way that users don't have to (re)start it ever - https://phabricator.wikimedia.org/T94883#1208219 (10yuvipanda) This is true now (pending announcement and documentation) because of T95210 [22:47:46] 10Tool-Labs, 3Labs-Q4-Sprint-3, 3ToolLabs-Goals-Q4: Make proxylistener not need to keep open socket connections open - https://phabricator.wikimedia.org/T96059#1208221 (10yuvipanda) So epilog scripts can be added on a per-queue basis and they do execute when killed OOM as well \o/ So now to fix proxylistene...