[00:52:31] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure: Diamond collected metrics about memory inaccurate until second reboot - https://phabricator.wikimedia.org/T91351#1080638 (10Krinkle) 3NEW
[00:52:51] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure: Diamond collected metrics about memory usage inaccurate until third reboot - https://phabricator.wikimedia.org/T91351#1080646 (10Krinkle)
[00:54:18] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure: Diamond collected metrics about memory usage inaccurate until third reboot - https://phabricator.wikimedia.org/T91351#1080638 (10Krinkle)
[02:33:07] <Magog_the_Ogre>	 the new hosts: which PHP version do they run?
[03:12:20] <legoktm>	 Magog_the_Ogre: trusty? 5.5 probably
[03:12:39] <Magog_the_Ogre>	 I hope so
[03:12:49] <Magog_the_Ogre>	 I am really sick of not having a finally block in PHP :)
[03:13:01] <Magog_the_Ogre>	 obviously not a big deal, just a personal preference
[03:14:04] <legoktm>	 they might also have hhvm, which identifies as 5.6
[03:16:25] <Magog_the_Ogre>	 I tried running hhvm, and it worked well until I updated to 5.6, and my XML parser broke
[03:16:39] <Magog_the_Ogre>	 thing is, my tool startup reads XML so...
[03:16:51] <Magog_the_Ogre>	 not sure if that was a PHP change or HHVM bug
[03:20:50] <Coren>	 Magog_the_Ogre: It may be either.  If you isolate the bug to HHVM, upstream is really cooperative.
[03:21:43] <Magog_the_Ogre>	 I find no amount of irony in the fact that Facebook, which has traditionally been a terribly written product, is producing one of the best ports available, on par with programming work by Google.
[03:22:32] <Coren>	 Necessity is the mother of invention.  They have several metric craploads of badly written PHP; it's easier to make the language faster than fix the codebase.  :-)
[03:22:55] <Coren>	 (Granted, but of that crappy PHP is third party, but still)
[03:22:59] <Coren>	 s/but/much/
[03:23:05] <Magog_the_Ogre>	 I think they've finally gotten better
[03:23:20] <Magog_the_Ogre>	 although their designers should be taken out back to the toolshed and shot
[03:23:33] <Magog_the_Ogre>	 their mobile app is one of the most infuriating things on the planet
[03:24:51] <Magog_the_Ogre>	 funny thing, 10s of billions of dollars gives you the ability to hire better programmers
[04:28:27] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: Puppetize /usr/local/bin/jobkill - https://phabricator.wikimedia.org/T90331#1081019 (10scfc) 5Open>3Resolved
[04:28:28] <wikibugs>	 10Tool-Labs: Make SGE be more informative about OOM kills - https://phabricator.wikimedia.org/T88824#1081020 (10scfc)
[04:46:40] <wikibugs>	 10Tool-Labs: create a node for dexbot service group which runs the user:dexbot - https://phabricator.wikimedia.org/T67199#1081031 (10scfc) 5Open>3Invalid I assume this is no longer needed; please reopen otherwise.
[05:10:52] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]  
[05:56:30] <wikibugs>	 10Tool-Labs: create a node for dexbot service group which runs the user:dexbot - https://phabricator.wikimedia.org/T67199#1081140 (10yuvipanda) In general I want us to move away from dedicated nodes. They aren't easy to distribute across nodes and are empty a lot of times. We should just increase capacity of gen...
[06:33:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[06:53:57] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]  
[07:03:54] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:11:38] <GerardM->	 WDQ is down again ?
[07:11:55] <GerardM->	 or is Tools ?
[07:12:04] <YuviPanda>	 GerardM-: wdq seems down, yeah.
[07:12:20] <GerardM->	 :(
[07:12:21] <YuviPanda>	 GerardM-: the processes are running at 100%, and I just restarted them, etc. so I’m not really sure what else I can do atm.
[07:12:30] <GerardM->	 I have a meeting in an hour I need it
[07:12:43] <GerardM->	 No I know
[07:12:51] <YuviPanda>	 I can restart it again but outside of that… nothing much I can do.
[07:13:10] <GerardM->	 how did the update go ?
[07:13:39] <GerardM->	 is catscan running ?
[07:13:48] <YuviPanda>	 GerardM-: I dunno anything about catscan
[07:14:05] <YuviPanda>	 anyway, I restarted wdq-mm again
[07:14:10] <GerardM->	 thanks
[07:24:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:27:47] <wikibugs>	 10Tool-Labs: Puppetize LVM extension for tools redis - https://phabricator.wikimedia.org/T91370#1081237 (10yuvipanda) 3NEW
[07:33:51] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis-slave is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0]  
[07:38:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis-slave is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:42:31] <werdna>	 hey, Special:NovaProxy isn't showing me anything anymore.
[09:05:54] <shinken-wm>	 RECOVERY - Puppet staleness on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [3600.0]  
[09:42:54] <wikibugs>	 10Wikimedia-Labs-Infrastructure, 10Beta-Cluster, 7Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#1081454 (10hashar)
[11:19:45] <YuviPanda>	 werdna: fixed
[11:19:53] <werdna>	 what did I do? :P
[11:20:15] <YuviPanda>	 werdna: Special:NovaProxy I mean
[11:20:26] <YuviPanda>	 !log projectproxy restarted dynamicproxy-api on dynamicproxy-gateway
[11:20:27] <labs-morebots>	 projectproxy is not a valid project.
[11:20:34] <YuviPanda>	 !log proxyproject restarted dynamicproxy-api on dynamicproxy-gateway
[11:20:34] <labs-morebots>	 proxyproject is not a valid project.
[11:20:38] <YuviPanda>	 !log proxy-project restarted dynamicproxy-api on dynamicproxy-gateway
[11:20:39] <labs-morebots>	 proxy-project is not a valid project.
[11:20:41] <YuviPanda>	 jesus
[11:21:13] <YuviPanda>	 !log project-proxy restarted dynamicproxy-api on dynamicproxy-gateway
[11:21:15] <labs-morebots>	 Logged the message, Master
[11:22:20] <wikibugs>	 6Labs, 10Wikimedia-Labs-wikitech-interface: Unable to see or delete existing web proxy - https://phabricator.wikimedia.org/T90391#1081600 (10yuvipanda) The proxy needed restarting and I have just done so. Is it still invisible for you?
[11:23:07] <werdna>	 got it
[11:24:18] <wikibugs>	 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, 10Tool-Labs-tools-Article-request, and 9 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1081601 (10yuvipanda)
[11:24:34] <wikibugs>	 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, 10Tool-Labs-tools-Article-request, and 9 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1031944 (10yuvipanda) I've edited the description a bit, please feel free to disagree / revert.
[11:25:16] <wikibugs>	 6Labs, 10Tool-Labs, 5Patch-For-Review: Retire 'tomcat' node, make Java apps run on the generic webgrid - https://phabricator.wikimedia.org/T91066#1081603 (10yuvipanda) Hmm, so there are still other tools that treat 'webgrid-tomcat' as they should treat webgrid-generic. I guess I should hunt them down one by...
[11:33:44] <YuviPanda>	 andrewbogott_afk: btw, proxies were dead. I restarted the api on dynamicproxy machine, and things are ok again
[11:33:48] <Kelson>	 YuviPanda: Hi, I come back to you regarding the public IP for mwoffliner3
[11:33:58] <YuviPanda>	 Kelson: aaah, right. sorry I didn’t get to it yesterday
[11:33:59] <YuviPanda>	 doing now
[11:34:33] <Kelson>	 YuviPanda: np, I still have a little bit storage on the VM...
[11:35:10] <YuviPanda>	 Kelson: right. I still reccomend using NFS for the storage, much cheaper than keeping spinning up new VMs :)
[11:35:18] <YuviPanda>	 !log mwoffliner increased public IP quota to 3
[11:35:20] <labs-morebots>	 Logged the message, Master
[11:36:41] <Kelson>	 YuviPanda: I should try this indeed. Will write my email to wmflabs-l today, I'm sure a few improvments proposition will be done.
[11:36:50] <YuviPanda>	 Kelson: yup, yup.
[11:37:49] <Kelson>	 YuviPanda: thx for the ip
[12:55:49] <ananthrk_>	 YuviPanda: Hey! Just wanted to let you know that I got the API working in tools
[12:55:57] <ananthrk_>	 thanks for the help yesterday
[12:59:52] <YuviPanda>	 ananthrk_: \o/ cool
[13:00:36] <ananthrk_>	 if I have to scale this API to support lot of data & users, should this be moved to labs?
[13:07:08] <YuviPanda>	 ananthrk_: depends on what you mean by ‘lot of data’ and ‘users'
[13:07:11] <YuviPanda>	 tools can do a lot :)
[13:08:26] <ananthrk_>	 hm..the current API is just a proof-of-concept idea built to understand the end-points & parameters required to expose clickstream data 
[13:09:13] <YuviPanda>	 ananthrk_: I can offer to help ‘scale it up’ when it hits current limits :)
[13:09:46] <ananthrk_>	 eventually this would be for clickstream data across all wiki pages in all languages. the data needs to be constantly updated as it is made available
[13:10:15] <ananthrk_>	 at the least, I would like to move away from SQLite and use mysql or postgres for storing the data
[13:10:29] <ananthrk_>	 is that going to be possible in tools?
[13:10:46] <YuviPanda>	 ananthrk_: yup. we have both mysql and postgres
[13:10:55] <YuviPanda>	 ananthrk_: if you look at your tools homedir, you will find replica.my.cnf
[13:11:07] <YuviPanda>	 ananthrk_: you can use that to connect to tools.labsdb host
[13:11:18] <ananthrk_>	 oh..good
[13:11:50] <YuviPanda>	 ananthrk_: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database
[13:11:57] <YuviPanda>	 postgres is available on an as-needed basis (for now)
[13:15:21] <ananthrk_>	 cool..thanks
[13:17:08] <ananthrk_>	 so if I want to create a separate database for this tool, do I create it in the same host?
[13:17:44] <YuviPanda>	 ananthrk_: you create it on tools-db, yeah.
[13:18:01] <YuviPanda>	 ananthrk_: the name of the database has to conform to the conventions described in the link I gave
[13:18:32] <ananthrk_>	 yeah..i see that it needs to be prefixed with the username specified in replica.my.conf
[13:18:36] <YuviPanda>	 yup
[13:19:57] <ananthrk_>	 "User-created databases can be created on the database hosting the replica servers or on a database local to the 'tools' project: tools-db"
[13:20:09] <ananthrk_>	 I can't understand the second part of this sentnce
[13:21:03] <ananthrk_>	 what do they mean by database local to the 'tools' project: tools-db? Does it mean one can instead directly create tables in this already existing database rather than create one on our own?
[13:21:21] <YuviPanda>	 ananthrk_: right, so you can create your databases in the *replicas* - so you can create them on say, enwiki.labsdb, etc. You shall do this if you want to perform cross db joins against enwiki, etc
[13:21:32] <YuviPanda>	 if not you create them on tools-db, which you can access with the DNS name tools.labsdb
[13:21:38] <YuviPanda>	 tools db docs kinda suck
[13:22:17] <petan>	 can someone tell me if http://stream.wikimedia.org/ is really down or it's just me?
[13:22:55] <petan>	 not by testing HTTP server but socket.IO
[13:23:10] <ananthrk_>	 oops...so tools.labsdb refer to the host then
[13:28:57] <YuviPanda>	 ananthrk_: yup.
[13:29:02] <YuviPanda>	 ananthrk_: it’s the host name
[13:29:13] <ananthrk_>	 got it..thanks
[13:29:47] <YuviPanda>	 ananthrk_: :) yw
[16:16:56] <andrewbogott>	 YuviPanda: every night I get some shinken warnings about puppet staleness on tools hosts, and then they recover… do you know what that’s about?
[16:17:07] <andrewbogott>	 Is it virt1000 getting overloaded?
[16:23:41] <greg-g>	 where's wikibugs?
[16:41:12] <legoktm>	 where's wikibugs??!?!?!
[16:41:37] <JohnFLewis>	 legoktm: gone away from its oppressive overlord
[16:42:29] <andrewbogott>	 I don’t know a thing about how wikibugs works, but I can restart if anyone can direct me how
[16:43:27] <legoktm>	 I'm doing that right now
[16:43:41] <andrewbogott>	 ‘k
[16:44:16] <ircnotifier>	 !log tools.wikibugs legoktm: Deployed 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge "Use brown instead of red for orange (=release) projects" wb2-irc
[16:44:18] <labs-morebots>	 Logged the message, Master
[16:44:49] <legoktm>	 the IRC process lost its connection to freenode but thought it was still connected...?
[16:45:22] <legoktm>	 also, maybe I'm blind but I can't find the quit message in my scrollback?
[16:45:27] <legoktm>	 hi wikibugs
[16:45:28] <legoktm>	 w/b
[16:45:42] <wikibugs>	 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1082223 (10Legoktm) !!
[16:45:47] <legoktm>	 yay
[16:49:47] <wikibugs>	 10Wikimedia-Labs-Infrastructure: Move LabsDB aliases and NAT to DNS and LabsDB servers - https://phabricator.wikimedia.org/T63897#1082228 (10yuvipanda) I think that ship sailed when those names were put in /etc/hosts so lets go with 2. That would also make it easier on other projects
[17:12:00] <^d>	 andrewbogott: Is https://phabricator.wikimedia.org/T90473 something you could take care of?
[17:12:18] <andrewbogott>	 ^d:Yep, I’ll do it right now
[17:12:59] <YuviPanda>	 ^d yay
[17:13:35] <^d>	 andrewbogott: Thanks!
[17:13:37] <andrewbogott>	 ^d: how’s that?
[17:14:23] <wikibugs>	 6Labs, 10Staging: Increase Security Groups quota on Wikitech staging project - https://phabricator.wikimedia.org/T90473#1082328 (10Andrew) 5Open>3Resolved a:3Andrew Should be done.
[17:14:54] <^d>	 Security Groups: 10/20
[17:14:59] <^d>	 Gives us plenty of room
[17:15:26] <wikibugs>	 10Tool-Labs, 7Tracking: make bigbrother or its replacement reliable - https://phabricator.wikimedia.org/T91414#1082331 (10JanZerebecki) 3NEW
[17:15:27] <andrewbogott>	 I don’t really know why there’s a quota for that anyway
[17:17:29] <wikibugs>	 10Tool-Labs: Test how bigbrother reacts to user names not resolving and, if necessary, fix it - https://phabricator.wikimedia.org/T90410#1082340 (10JanZerebecki)
[17:17:30] <wikibugs>	 6Labs, 10Tool-Labs: Have bigbrother run on multiple nodes to provide redundancy against tools-submit failure - https://phabricator.wikimedia.org/T91237#1082341 (10JanZerebecki)
[17:17:31] <wikibugs>	 6Labs, 10Tool-Labs: Monitor bigbrother - https://phabricator.wikimedia.org/T90850#1082342 (10JanZerebecki)
[17:23:40] <^d>	 andrewbogott: can't have too much security :)
[17:27:35] <mutante>	 wikibugs: help
[17:27:45] <mutante>	 wikibugs: why are you only here but left all the other channels
[17:28:10] <wikibugs>	 6Labs, 10Tool-Labs: Monitor toollabs home page to make sure it is up - https://phabricator.wikimedia.org/T90847#1082393 (10Jdforrester-WMF)
[17:28:11] <wikibugs>	 10Tool-Labs: Setup (and document) an easy way to run nodejs based tools - https://phabricator.wikimedia.org/T1102#1082391 (10Jdforrester-WMF)
[17:28:14] <wikibugs>	 10Tool-Labs: Job labs-toollabs-debian-glue fails - https://phabricator.wikimedia.org/T90790#1082395 (10Jdforrester-WMF)
[17:28:40] <wikibugs>	 10Tool-Labs-tools-DrTrigonBot---subster: DRTRIGON-132 Second stage of substitutions for metadata info - https://phabricator.wikimedia.org/T61391#1082419 (10Jdforrester-WMF)
[17:28:44] <wikibugs>	 10Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1082422 (10Jdforrester-WMF)
[17:29:02] <wikibugs>	 6Labs: Monitor that nfs-manage-volumes deamon is running on the NFS hosts - https://phabricator.wikimedia.org/T88664#1082442 (10Jdforrester-WMF)
[17:29:04] <wikibugs>	 6Labs: Make sure that manage-nfs-volumes-deamon can not run as root - https://phabricator.wikimedia.org/T88579#1082440 (10Jdforrester-WMF)
[17:29:05] <wikibugs>	 6Labs: Fix puppet code to make sure that manage-nfs-volumes deamon is running - https://phabricator.wikimedia.org/T88669#1082441 (10Jdforrester-WMF)
[17:29:09] <wikibugs>	 10Wikibugs: wikibugs has too many channels, causing join flood - https://phabricator.wikimedia.org/T86758#1082448 (10Jdforrester-WMF)
[18:50:18] <wikibugs>	 6Labs: labs_lvm can clash file resource for mount point with other packages - https://phabricator.wikimedia.org/T91225#1083404 (10coren) Unless I'm mistaken, that would work //iff// there already is a resource for the mount point (which there generally will not be).  Perhaps remove the file resource from labs_lv...
[19:51:35] <wikibugs>	 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, 10Tool-Labs-tools-Article-request, and 9 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1083850 (10Andrew) I think that tools and beta tickets should probably /not/ add labs by default; those tasks which bubble...
[20:01:14] <wikibugs>	 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1083876 (10Legoktm) !!!
[20:02:40] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge "Use brown instead of red for orange (=release) projects"
[20:02:45] <labs-morebots>	 Logged the message, Master
[20:03:21] <wikibugs>	 10Wikibugs, 6operations: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1083884 (10Legoktm) !!!
[20:03:37] <wikibugs>	 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1083887 (10Legoktm)
[20:11:21] <matanya>	 Coren: me again 
[20:11:34] <Coren>	 matanya: Indeed.
[20:11:44] <matanya>	 https://phabricator.wikimedia.org/T90437
[20:11:59] <matanya>	 why do you keep conntrack in the first place ?
[20:12:30] <Coren>	 matanya: That's how nova-network does its thang.
[20:12:45] <matanya>	 so it is a must ?
[20:13:00] <matanya>	 I wish we could use neutron
[20:14:27] <Coren>	 matanya: I don't think it's change anything: neutron also has to implement security groups
[20:14:43] <Coren>	 matanya: And even if it didn't, the natting demands it.
[20:14:51] <matanya>	 I see
[20:15:05] <matanya>	 In prod we don't care :/
[20:15:27] <Coren>	 Prod also doesn't do natting (thank $deity)
[20:15:50] <Coren>	 There's already a ticket for monitoring this.  Lemme try to find it.
[20:16:18] <matanya>	 the one linked above
[20:17:18] <Coren>	 Oh duh, I though that's one you just added, I was looking to merge it.  :-)
[20:18:24] <Coren>	 matanya: It's on my todo; but not urgently since we're sticking way below the maximum atm (I've increased it by two orders of magnitude)
[20:18:54] <matanya>	 i'll try to write a lame thing now
[20:20:14] <Coren>	 matanya: If you do, get the numbers from /proc/sys - the sysctl interface has been deprecated in ages.
[20:20:41] <Coren>	 Since 2.6.24 at least.
[20:20:41] <matanya>	 i'll just use conntrack, if it is still suppporteed
[20:21:27] <Coren>	 It's probably simpler to read the files in /proc as well as avoids shelling out
[20:21:43] <matanya>	 makes sense
[20:22:30] <Coren>	 Both contain just a number on a line: /proc/sys/net/netfilter/nf_conntrack_{max,count}
[20:22:56] <matanya>	 yes
[20:23:30] <matanya>	 ah, i don't have access to labnet1001
[20:23:52] <matanya>	 right, forgot that. what is the max and count now ?
[20:24:07] <Coren>	 76189 / 262144 atm
[20:26:05] <matanya>	 Coren: does 15% warn and 10% crit work ?
[20:26:30] <Coren>	 Might even warn at 20%; we almost certainly don't want to have it this close as a rule.
[20:27:53] <matanya>	 ok, and it uses shinken? would monitor_service work ?
[20:31:08] * brion wanders into his labs-vagrant instance to update it. yay labs!
[20:32:32] <Coren>	 matanya: I suppose it would.  I honestly haven't looked at it yet.
[20:33:05] <matanya>	 ok, i'll shoot in the dark a bit
[21:01:53] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0]  
[21:04:43] <Ironholds>	 Coren, so, the web proxy problem; is there any ETA on fixing that?
[21:05:33] <Coren>	 Ironholds: I plan on corndering andrewbogott this evening so that we can work on it.  I've taken a look at the osm code and got confused.
[21:05:53] <Ironholds>	 cool; thanks. It's a blocker on spinning up some visualisations Erik wanted.
[21:05:55] * Coren isn't sure what "corndering" means, but it sounds suitable.
[21:13:36] <andrewbogott>	 Coren: what’s up?
[21:13:52] <Coren>	 Oh hey, you're around early today!
[21:14:45] <Coren>	 andrewbogott: https://phabricator.wikimedia.org/T91114 basically.
[21:15:08] <Coren>	 Although the "Incidentally" part is probably the thing you already fixed with the in-addr.arpa thing.
[21:15:27] * andrewbogott looks
[21:15:54] * Coren tries creating a proxy now to make sure that's still buggy.
[21:17:01] <Coren>	 Yeah, the Incidentally part no longer is true.
[21:17:26] <Coren>	 ... what.
[21:17:27] <andrewbogott>	 Hm, just tried and it worked for me
[21:17:38] <Coren>	 Ironholds: It just worked for me today.
[21:17:46] <andrewbogott>	 Maybe it was the same bug?  Although I don’t know how or why
[21:17:53] <Ironholds>	 Coren, huh! Let me try again
[21:17:55] <Coren>	 andrewbogott: I hadn't tested it since it you fixed the other bug.
[21:18:41] <Ironholds>	 holy crap it worked
[21:18:47] <Ironholds>	 labs is /basically/ voodoo
[21:18:49] <Coren>	 andrewbogott: But it would sorta make sense since they both used the domain list; it'd make sense that they could both be affected.
[21:18:53] <Ironholds>	 lots of chanting and precise timing
[21:19:05] <andrewbogott>	 yeah
[21:19:06] <Coren>	 Ironholds: No, I think that's Rock Band 3.
[21:19:33] <Ironholds>	 fair
[21:20:01] <wikibugs>	 6Labs, 10Wikimedia-Labs-wikitech-interface: Proxy creation fails with opaque error message - https://phabricator.wikimedia.org/T91114#1084354 (10coren) 5Open>3Resolved a:3coren This appears to have been fixed by a related (to DNS) issue being fixed.
[21:20:22] <Coren>	 andrewbogott: I should have tried that again after you fixed something with a similar buglet.
[21:20:39] <Coren>	 Although Ironholds would not have been remiss to test it as well.  :-P
[21:20:47] <Ironholds>	 I did test it!
[21:20:51] <Ironholds>	 yesterday! It blew up
[21:22:20] <Coren>	 After https://phabricator.wikimedia.org/T90856 was fixed?
[21:22:43] * Coren hates heisenbugs.
[21:23:54] <andrewbogott>	 Coren: thanks to memcached it took 24 hours for the ldap change to affect wikitech
[21:23:55] <wikibugs>	 6Labs, 10Wikimedia-Labs-wikitech-interface: Proxy creation fails with opaque error message - https://phabricator.wikimedia.org/T91114#1084365 (10coren) (For the record, that was T90856)  A further note, while the issue is not present now @ironholds reports that it may have occured as late as 2015-03-02
[21:24:05] <andrewbogott>	 So it could still  have been broken early yesterday… maybe…
[21:24:20] <andrewbogott>	 Hm, nope
[21:24:20] <Ironholds>	 I'm talking, like, 2pm EST yesterday
[21:24:26] <Ironholds>	 ask milimetric!
[21:24:46] <Coren>	 Ironholds: I'm pretty sure we believe you on your say-so.  :-)
[21:25:09] <Ironholds>	 ;p
[21:25:21] <Ironholds>	 now to try and work out why shiny, which is listening on 3838, can't do anything with the proxy
[21:25:23] * Ironholds strokes beard
[21:26:07] <andrewbogott>	 Ironholds: the proxy only relays 80 and 443
[21:26:31] <Ironholds>	 oh bollocks
[21:26:48] * Ironholds will play with it. Thanks.
[21:27:06] <Cyberpower678>	 Ironholds, I have never seen you on IRC. :p
[21:27:23] <Ironholds>	 Cyberpower678, I'm on IRC regularly and have been since the age of nine.
[21:27:29] <Cyberpower678>	 :O
[21:27:38] <Cyberpower678>	 Are you regularly in this channel?
[21:27:57] <Ironholds>	 nope
[21:28:06] * Coren points out that this means he's been online for, what, 5 years?
[21:28:17] * Coren ducks.
[21:28:31] <Ironholds>	 Coren, and grandpa's joke about me being a foetus is only off by a decade!
[21:28:41] <Ironholds>	 you're getting better! Normally when the fog descends you're at /least/ 20 years off :)
[21:28:56] <Cyberpower678>	 You're 34?
[21:29:02] <Ironholds>	 24 ;p
[21:29:13] <Ironholds>	 Coren, I'll tell your nurse you can be trusted with some extra rusks this month ;p
[21:29:14] <Cyberpower678>	 ...
[21:29:27] <Ironholds>	 Cyberpower678, (9 + 5 + 10)
[21:29:34] <Coren>	 Ironholds: Oh, no worries, talking about IRC doesn't make me feel any younger for realz.  I was using IRC when there was /a/ network.  :-)
[21:29:39] <Cyberpower678>	 On since 9, on for 5 years, missed by 20 = 34
[21:29:56] <Ironholds>	 except I said he missed by a decade, which was better than usually, when he'd miss by two.
[21:30:11] <Cyberpower678>	 you're at /least/ 20 years off :)
[21:30:11] <Cyberpower678>	 you're at /least/ 20 years off :)
[21:30:12] <Cyberpower678>	 you're at /least/ 20 years off :)
[21:30:12] <Cyberpower678>	 you're at /least/ 20 years off :)
[21:30:12] <Cyberpower678>	 you're at /least/ 20 years off :)
[21:30:13] <Cyberpower678>	 :O
[21:30:22] <Ironholds>	 "normally...you're at least".
[21:30:29] <Cyberpower678>	 I didn't mean to do that. :p
[21:30:38] <Ironholds>	 and I'm done having a long explanation about what I meant when I said a thing you didn't grok ;p
[21:30:49] <Cyberpower678>	 :p
[21:30:50] <Ironholds>	 Coren, I had someone tell me the other day that their first computer was a Windows XP machine
[21:30:57] <Ironholds>	 to which my response was: how are you even old enough to talk.
[21:30:58] <Cyberpower678>	 Same
[21:31:13] <Ironholds>	 then they pointed out that my hair was full of greys and I had a widow's peak thing going on and I retired to the home.
[21:31:19] <Cyberpower678>	 My fisting computer was an XP machine
[21:31:35] <Cyberpower678>	 I'm 21
[21:31:40] <mutante>	 nice autocomplete 
[21:31:51] <Cyberpower678>	 *first.  OMG
[21:31:53] <Ironholds>	 ...
[21:32:25] <Cyberpower678>	 So I got my first computer at the age of 8
[21:32:29] <Coren>	 Heh.  My first computer was a Southwest S09 built from kit.  :-)
[21:33:00] <Ironholds>	 the first one I ever owned just for me ran windows ME
[21:33:02] <Coren>	 http://www.swtpc.com/mholley/swtpc_6809.htm
[21:33:04] <Ironholds>	 my parents were MONSTERS.
[21:33:11] * Cyberpower678 needs to get back to studying so he can fail his EE 350 exam.
[21:33:15] <Coren>	 Ironholds: At least it wasn't Bob.  :-)
[21:34:14] <mutante>	 My first computer was a Commodore 128. They told my mom it was the next big thing after the C64.
[21:34:29] <Cyberpower678>	 It technically was. ;P
[21:34:59] <wikibugs>	 6Labs: labs_lvm can clash file resource for mount point with other packages - https://phabricator.wikimedia.org/T91225#1084397 (10scfc) I think you're right.  We probably need to use ensure_resource() instead.  (Ceterum censeo: Moving partitions around in a running system is black magic :-).  OpenStack should pr...
[21:35:12] <Cyberpower678>	 mutante, It's all about the Pentiums by Weird Al
[21:35:16] <Cyberpower678>	 Play it
[21:35:57] <mutante>	 Cyberpower678: sprdef was the shit
[21:36:20] <mutante>	 built-in editor for sprites
[21:36:38] <Cyberpower678>	 :p
[21:36:55] <mutante>	 but besides that, always just held the Commodore key to boot into C64 mode.. i mean like what > 10,000 games for C64 and 3(!) for 128 mode
[21:37:55] <JohnFLewis>	 mutante: :p
[21:38:57] <JohnFLewis>	 Those 3 must have been pretty damn good :p
[21:42:41] <mutante>	 JohnFLewis: omg, " Hitchhiker's Guide to the Galaxy" and i had no idea then
[22:09:08] <wikibugs>	 6Labs, 10Wikidata: wdq.wmflabs.org does not update (data week old) - https://phabricator.wikimedia.org/T89583#1084584 (10Johansen.fred)
[22:09:59] <Cyberpower678>	 YAY.  Exams cancelled.
[22:10:04] <Cyberpower678>	 Due to weather.
[22:10:09] <Ironholds>	 Coren, good news!
[22:10:30] <Coren>	 Ironholds: What news are that?
[22:10:32] <Cyberpower678>	 Ironholds, now I can annoy you some more. XD
[22:10:36] <Ironholds>	 you know the proxy bug that's fixed?
[22:10:38] <Ironholds>	 it's not fixed
[22:10:40] <Ironholds>	 https://gist.github.com/ottomata/4e0193bb8b2f46f8b861
[22:10:50] <Ironholds>	 doesn't seem to be hitting old proxies, but newly-created ones, bbzzzzzt
[22:11:02] <Ironholds>	 Cyberpower678, I mean, you could alternately find something more useful to humanity than annoying people
[22:11:17] <Coren>	 Ironholds: Wait, you can't access the proxy from inside labs.
[22:11:30] <Ironholds>	 can't access the proxy externally either
[22:11:30] <Cyberpower678>	 Ironholds, until I become a legit Electrical Engineer, annoying people is all I'll do. :p
[22:11:41] <Ironholds>	 go to http://datavis.wmflabs.org and see if it works
[22:11:58] <Ironholds>	 Cyberpower678, really? Because I know a good 20-30 EE, CE and CS majors and they're all really productive people.
[22:12:21] <Ironholds>	 if you have an actual question, feel free to ask me, but no, I do not want you to interact with me if your /intent/ is, from the get-go, to annoy.
[22:13:10] <Cyberpower678>	 Ironholds, my intent is to never annoy.  I was only joshing.
[22:13:18] <Ironholds>	 then yes, feel free to ask questions
[22:13:41] <Coren>	 Ironholds: That has nothing to do with the bug.  What the dynamic proxy does is set up a DNS entry - which is there - and set a mapping for the proxy itself - which is also there.  What /isn't/ there is shiny actually listening on its external port 3838.  :-)
[22:13:47] <Cyberpower678>	 And I always try to be productive.  Else I would be blocked on Wikipedia already.
[22:13:48] <Coren>	 marc@tools-login:~$ telnet shiny 3838
[22:13:48] <Coren>	 Trying 10.68.17.20...
[22:13:54] <Coren>	 :-)
[22:14:11] <Ironholds>	 Coren, that's because I just set it to 8080 to reduce the number of variables (there are existing proxies on 8080. There were none on 3838)
[22:14:32] <Cyberpower678>	 Who uses 3838?
[22:15:06] <Cyberpower678>	 @seen nakon
[22:15:06] <wm-bot>	 Cyberpower678: Last time I saw nakon they were quitting the network with reason: no reason was given N/A at 2/27/2015 4:31:03 AM (4d17h44m3s ago)
[22:15:06] <Coren>	 Ironholds: Remind me the project name?
[22:15:17] <Cyberpower678>	 MusikAnimal, you there?
[22:16:56] <Ironholds>	 Coren, shiny.eqiad.wmflabs or datavis.wmflabs.org
[22:17:23] <Ironholds>	 Cyberpower678, exactly
[22:17:43] <Ironholds>	 it's not used for a consistent service, unlike, say, 80 or 443, so systems can use it without running into problems.
[22:19:29] <Coren>	 Ironholds: I really see nothing wrong with the proxy.  What is supposed to be different?
[22:19:41] <Coren>	 Ironholds: "Welcome to Shiny Server!" and "It's Alive!" seem to bode well.
[22:19:58] <MusikAnimal>	 hi
[22:21:43] <Ironholds>	 Coren, holy crap it works
[22:21:53] <Ironholds>	 that is the weirdest thing. I swear to god it wasn't working 10 minutes ago
[22:21:54] <Coren>	 That's the bug, is it?
[22:22:09] <Coren>	 "It works and it wasn't supposed to!"
[22:22:51] <Coren>	 Ironholds: I dunno shiny.  Maybe it takes some time to wind up?
[22:23:24] <Ironholds>	 well, I spawned the initial instance an hour ago. /shrug
[22:23:28] <Ironholds>	 it works, is the important thing!"
[22:25:41] <Cyberpower678>	 MusikAnimal, can you get nakon's attention for me.
[22:26:31] <MusikAnimal>	 I don't know if there's anything I can do
[22:27:01] <MusikAnimal>	 on-wiki his last edit was 2:17 this morning GMT
[22:27:25] <MusikAnimal>	 try email?
[22:28:40] <Cyberpower678>	 MusikAnimal, should I just go ahead and add him, or should you and I talk to him first?
[22:29:02] <MusikAnimal>	 you've talked him haven't you?
[22:29:29] <Cyberpower678>	 He's offered, but I would ideally like to talk to him and see you talk to him before adding him.
[22:29:47] <MusikAnimal>	 well let's wait till he's back online, or shoot him an email and cc me
[22:30:21] <Cyberpower678>	 MusikAnimal, I can't CC in a Wikipedia email. :/
[22:34:21] <legoktm>	 if a job is running, is it possible to get the parameters passed to jsub to start it? 
[22:34:37] <legoktm>	 nvm, found it in .bash_history :P
[22:39:00] <Cyberpower678>	 MusikAnimal, sent an email to him.
[22:41:27] <MusikAnimal>	 +1