[01:07:22] <Guerillero>	 Coren, are you around
[02:10:25] <andrewbogott>	 matanya: how is your new, giant video instance faring?  Staying up and such?
[05:42:25] <matanya>	 andrewbogott: sweet and charming, thanks!
[06:09:35] <andrewbogott>	 matanya: you already killed off the older, smaller instance, right?
[06:40:19] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[07:04:50] <matanya>	 I haven't andrewbogott I can i f you wish
[07:05:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:05:18] <andrewbogott>	 matanya: Keep it if it's useful; if you're only using one at a time, though, please delete the unused instance
[08:09:11] <matanya>	 done andrewbogott 
[08:09:18] <andrewbogott>	 thanks!
[09:51:58] <hashar>	 YuviPanda: good afternoon.  Do you know whether the labs DNS issue has been figured out ? :]
[09:52:06] <hashar>	 else will poke coren about it this afternoon
[09:52:10] <YuviPanda>	 hashar: waiting for Coren mostly, yeah
[09:52:11] <YuviPanda>	 so no.
[09:52:24] <YuviPanda>	 hashar: I see that #wikimedia-qa has been flooded
[09:52:36] <hashar>	 YuviPanda: my mailbox as well
[09:52:42] <YuviPanda>	 yeah that too
[12:53:44] <Tobi_WMDE_SW>	 heya everybody, anyone an idea why http://wikidata.beta.wmflabs.org/ cannot be reached?
[12:57:37] <YuviPanda>	 !log icinga delete icinga.wmflabs.org, was confusing people. Will set up redirect to shinken at some point
[12:57:41] <labs-morebots>	 Logged the message, Master
[12:57:46] <YuviPanda>	 Tobi_WMDE_SW: ugh, hoo was just pointing that out. 
[12:57:50] <YuviPanda>	 Tobi_WMDE_SW: unsure, am looking around.
[12:58:00] <Tobi_WMDE_SW>	 YuviPanda: thanks!
[12:58:02] <YuviPanda>	 Tobi_WMDE_SW: do you know when it stopped working? 
[12:58:09] <YuviPanda>	 Tobi_WMDE_SW: can you file a ticket in phab?
[12:59:44] <Tobi_WMDE_SW>	 YuviPanda: looking at the browsertest jobs it seems it's broken since dec 30th already.. :(
[12:59:50] <YuviPanda>	 hmm, sigh
[13:02:23] <Tobi_WMDE_SW>	 YuviPanda: is the "beta-cluster" project the one wher I should file the bug? https://phabricator.wikimedia.org/project/view/497/
[13:02:31] <YuviPanda>	 Tobi_WMDE_SW: yah
[13:06:16] <Tobi_WMDE_SW>	 YuviPanda: https://phabricator.wikimedia.org/T85793
[13:07:17] <YuviPanda>	 thanks
[13:12:48] <YuviPanda>	 Tobi_WMDE_SW: updated, and works now :) 
[13:12:52] <YuviPanda>	 well
[13:12:57] <YuviPanda>	 once DNS cache clears, I hope
[13:15:30] <Tobi_WMDE_SW>	 YuviPanda: wow, great! thanks!
[13:15:40] <YuviPanda>	 Tobi_WMDE_SW: can you try again?
[13:17:25] <Tobi_WMDE_SW>	 YuviPanda: hm, not yet working
[13:18:21] <YuviPanda>	 Tobi_WMDE_SW: https://phabricator.wikimedia.org/P193 on a host without the DNS cached
[13:18:29] <YuviPanda>	 Tobi_WMDE_SW: so it’s probably your cache that’s still stale. give it a bit
[13:19:22] <Tobi_WMDE_SW>	 YuviPanda: ok, I'll just give it a bit more time
[13:24:24] <Tobi_WMDE_SW>	 YuviPanda: so, the browsertest just passed, so I think it works. thanks again!
[13:24:43] <YuviPanda>	 Tobi_WMDE_SW: yw! I’ll resolve the ticket now
[13:25:34] <Tobi_WMDE_SW>	 can be resolved
[13:53:16] <wikibugs>	 3Continuous-Integration, Wikimedia-Labs-Infrastructure: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#954959 (10hashar) To workaround the boot sequence not finding jenkins-slave user, we could have the tmpfs mounted ju...
[14:37:18] <wikibugs>	 3Labs-Team: Fix dnsmasq in nova to not cache - https://phabricator.wikimedia.org/T80293#955049 (10coren) p:5Normal>3High
[14:56:26] <wikibugs>	 3Continuous-Integration, Wikimedia-Labs-Infrastructure: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#955074 (10hashar) Applied on all labs CI slaves using: ``` umount /mnt/home/jenkins-deploy/tmpfs rmdir /mnt/home/jen...
[15:07:33] <Coren>	 andrewbogott_afk: Just emailed you about the lvm mess
[15:10:23] <T13|mobile>	 Cyberpower678: have you gotten access to those tools you asked Coren about yet?
[15:10:54] <Cyberpower678>	 T13|mobile, he responded on the mailing list.  He said to give him a day or 2.
[15:11:29] <Coren>	 Yeah, we need to sort out how those things are to be done (there is no question that this is a case where it makes sense to do so).
[15:11:54] <T13|mobile>	 Cool. :)
[15:13:17] <hashar>	 Coren: good morning and happy new year :-]
[15:13:37] <hashar>	 Coren: I have a couple things for you whenever you have finished your breakfast
[15:13:43] <Cyberpower678>	 T13|mobile, xTools is hung up.  Do you think you can use that to your advantage to figure out the cause?
[15:13:44] <Coren>	 hashar: You too.  :_)
[15:13:56] <Coren>	 hashar: If it's about DNS, that's my priority today.  :-P
[15:14:13] * hashar strikes an item
[15:14:23] <hashar>	 I had another one but it is proving to not be working properly haha
[15:15:03] <Krinkle>	 Coren: I'm trying to figure out why every 4-6 months mw-snapshots tool is broken because the periodic job (which takes about 10 minutes normally and runs every hour) suddenly stops running (its still starts hourly but doesn't finish).
[15:15:06] <Technical_13>	 Coffee just finished brewing...
[15:15:09] <Krinkle>	 https://tools.wmflabs.org/snapshots/?action=updatelog
[15:15:25] <Krinkle>	 Coren: example run qacct -j 7066247; Not sure what I'm looking for or why it stopped.
[15:16:03] <Coren>	 Krinkle: Interesting.  Did you notice a pattern about when it fails?
[15:16:32] <Krinkle>	 Coren: If I run the script on tools-login now without jsub it will succeed in 10 minutes and then the next jsub will run normally for another few months.
[15:16:34] <Cyberpower678>	 Technical_13, tools has seemingly hung up.  Do you think you can use that to see what's causing the hangup?
[15:16:38] <Cyberpower678>	 *xTools
[15:16:39] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#955102 (10Dzahn) why is FTP actually needed? as soon as an SSH is running anyways you can already SCP and there are GUI clients for SCP (WinSCP e...
[15:16:51] <Krinkle>	 Coren: However as of a few days ago, every run fails. It hasn't produced a tar.gz in over 250 hours.
[15:17:05] <Technical_13>	 Cyberpower678: I'm not sure.  I haven't spent any time figuring out how to SSH into tools yet.
[15:17:08] <Technical_13>	 I've been too busy and focused on https://test.wikipedia.org/wiki/User:Technical_13/SpamUNblocker.js
[15:17:12] <Coren>	 YuviPanda: There's a whole lot of bikeshedding about what is a proper solution to the DNS problems but right now I'm concerned about solving the immediate issue only.  Ima create a recusror as previously discussed and point instances at it instead of dnsmasq.
[15:17:18] <Krinkle>	 Coren: If I run the script on tools-login now without jsub I know it will succeed in 10 minutes (just like a few months ago when this happened as well) and then the next jsub will run normally for another few months.
[15:17:19] <Technical_13>	 And I haven't had morning coffee.
[15:17:36] * Technical_13 runs off to kitchen...
[15:17:54] <Krinkle>	 It's probably a bug in git or my script, but I'd have to know why it stopped to know where to look for debugging. E.g. time out, or cpu/memory, or something else.
[15:18:06] <YuviPanda>	 Coren: yeah, I suspect a lot of instability is due to this, and fixing the immediate issue would give us breathing room
[15:18:08] <Cyberpower678>	 SSH into tools-login.wmflabs.org
[15:18:14] <Cyberpower678>	 become xTools
[15:18:35] <Coren>	 Krinkle: ... what.  Huh.  Lemme try to see if there's an obvious point where it got stuck.
[15:19:09] <wikibugs>	 3Wikimedia-Labs-Infrastructure: make Debian Jessie image for labs - https://phabricator.wikimedia.org/T75592#955104 (10Dzahn) >>! In T75592#793510, @Andrew wrote: > Unfortunately, the software that we use to build official labs images (python-vm-builder) doesn't support Debian.   ahh.. i see. thanks for the expl...
[15:19:10] <Coren>	 Krinkle: Do you have a currently hung job?
[15:19:36] <Technical_13>	 Cyberpower678: link to the instructions for loging into tools with PuTTY?
[15:21:29] <Cyberpower678>	 Technical_13, https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP
[15:22:21] <Technical_13>	 Cyberpower678: USERNAME is my shell name, which is case sensitive if I remember correctly, yes?
[15:26:59] <Cyberpower678>	 Yep.
[15:27:14] <Krinkle>	 Coren: The job id (7066247) is the last one I ran. There has been one other run after it from cron/jsub (triggered 27 minutes ago).
[15:27:23] <Krinkle>	 Coren: They don't hang (at least, they don't stay around in qstat)
[15:27:41] <Coren>	 But they also don't work?
[15:33:34] <Technical_13>	 Cyberpower678: I'm in... Is there an xTools-dev channel?
[15:33:48] <Cyberpower678>	 No.
[15:33:50] <Technical_13>	 So you can show me around without disrupting this channel?
[15:33:55] <Technical_13>	 ##T13 then?
[15:44:02] <Cyberpower678>	 Coren, ##T13
[15:46:08] <hashar>	 Coren: about the DNS, a few months ago we had a bot spamming hundreds of queries per seconds on beta cluster
[15:46:32] <hashar>	 Coren: and MediaWiki ends up doing a lot of DNS requests which ended up saturating dnsmasq :/  Might be the case again
[15:47:26] <Coren>	 hashar: That's always a possibility, but it's a symptom and not the cause.  Having a DNS server fail intermitently if you look at it too hard is a Big Problem.
[15:48:10] <hashar>	 Coren: yeah I agree
[15:48:47] <hashar>	 I found out the issue by looking in ganglia at the # of packets on the network interface
[16:11:07] <wikibugs>	 3Labs-Team: Increase storage available to labs NFS server - https://phabricator.wikimedia.org/T85607#955265 (10mark) What's the status of this order? Perhaps link to the RT ticket?
[16:12:35] <wikibugs>	 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#955276 (10mark) 24 hours of downtime is very painful obviously. Any way we could do it faster, or avoid it altogether?  When do you anticipate doing this? We'll need to plan and communicate about it well in advance...
[16:17:18] <wikibugs>	 3Beta-Cluster, Labs-Team, operations: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#955291 (10greg) >>! In T1259#943783, @yuvipanda wrote: > Also, all of these are hhvm - are the hhvm core dumps from beta useful at all, or should we disable them?  @joe or @ori or @bd808 ?
[16:18:36] <wikibugs>	 3Labs-Team: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#955296 (10coren) I've already started the discussion on labs-l about scheduling it; and revised the tentative schedule to address concerns from the beta team.  I've done some dry run, and we may be able to do so with as...
[16:21:04] <wikibugs>	 3Beta-Cluster, Labs-Team, operations: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#955312 (10bd808) >>! In T1259#955291, @greg wrote: >>>! In T1259#943783, @yuvipanda wrote: >> Also, all of these are hhvm - are the hhvm core dumps from beta useful at all, or should we disab...
[16:26:17] <wikibugs>	 3Labs-Team: Increase storage available to labs NFS server - https://phabricator.wikimedia.org/T85607#955319 (10coren) This is tracked by [[ https://rt.wikimedia.org/Ticket/Display.html?id=8830 | RT 8830 ]] although a better description of the status is "ordering process in progress" it seems; I see no activity a...
[16:27:05] <Coren>	 YuviPanda: All your ipblocks are belong to... oh, nevermind.  Done.  :-)
[16:33:14] <YuviPanda>	 Coren: doesn’t seem to have changed it anywhere...
[16:33:23] <YuviPanda>	 at least on labsdb1001
[16:33:37] <Coren>	 Hm?
[16:33:40] * Coren checks.
[16:35:23] <Coren>	 Huh.  That's odd.
[16:36:18] <Coren>	 Yuvi... dude.  You didn't, you know, *commit* the change.
[16:36:19] <YuviPanda>	 yeah, on labsdb1002 too
[16:36:41] <Coren>	 So my git pull didn't bring it in.  :-)
[16:36:59] <Coren>	 s/commit/submit/ (proper terminology)
[16:37:41] <Coren>	 So what I did was a really expensive noop.  :-)
[16:37:53] <YuviPanda>	 heh
[16:39:16] <wikibugs>	 3Continuous-Integration, Wikimedia-Labs-Infrastructure: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#955365 (10Krinkle) 5Open>3Resolved Thanks!  By the way, do we have a ticket to track LDAP not being available? O...
[16:42:59] <Krinkle>	 Coren: The jobs just stop at some point. I'm trying to figure out why they are aborted. While in the middle of running various parts of the job still.
[16:43:20] <Krinkle>	 I've looked at the qacct entry but not getting anywhere.
[16:47:33] <Coren>	 You'll proably need to add some runtime logging to help.
[16:47:53] <Coren>	 The problem with scheduled jobs is that they tend to run while you're not watching them.  :-)
[16:52:14] <Krinkle>	 Coren: They stop in the middle of a php shell exec running a git command 'git fetch origin'
[16:52:18] <Krinkle>	 exit code 130
[16:52:38] <Krinkle>	 the script gets aborted by the grid I presume.
[16:57:53] <YuviPanda>	 Coren: did you run the script again?
[17:00:46] <Coren>	 ... you still haven't submitted the change.
[17:01:10] <Coren>	 Krinkle: 130 is almost certainly out of memory
[17:01:45] <YuviPanda>	 Coren: oh
[17:01:46] <YuviPanda>	 Coren: damn
[17:02:08] <YuviPanda>	 inconsistent jenkins
[17:02:15] <YuviPanda>	 Coren: submitted!
[17:03:32] <Coren>	 Running again.
[17:06:57] <marktraceur>	 incontinent jenkins
[17:11:24] <Technical_13>	 What do the `state` flags ina qstat result represent? I'm guessing r=running and qw=queried work?
[17:12:45] <wikibugs>	 3Labs-Team, Beta-Cluster, operations: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#955410 (10greg) @Ori, ideas on how to manage these? Should you (or someone else close to HHVM) take a weekly gander at the dumps on beta or something else? Ideally we'd have auto bug reportin...
[17:13:16] <Coren>	 Technical_13: qw waiting in queue
[17:13:38] <Coren>	 Or, more precisely, 'queued, waiting'
[17:13:47] <wikibugs>	 3Labs-Team, operations: stray files created in /etc/ssh/userkeys - https://phabricator.wikimedia.org/T85814#955415 (10fgiunchedi) 3NEW
[17:13:51] <Technical_13>	 Close enough .:)
[17:23:19] <negative24>	 Which puppet role include php5-cli?
[17:24:11] <YuviPanda>	 negative24: the LAMP role includes that, I think.
[17:24:16] <YuviPanda>	 do you need *just* php5-cli?
[17:25:10] <negative24>	 I'm trying to run update.php on a new LAMP instance and I have role::lamp::labs installed but not php5-cli
[17:25:29] <negative24>	 I think php is installed as an apache module
[17:26:30] <YuviPanda>	 negative24: hmm, let me look
[17:27:12] <YuviPanda>	 negative24: indeed, you’re right. it doesn’t seem to be installe.d
[17:27:15] <YuviPanda>	 negative24: let me fix that
[17:29:41] <negative24>	 yup, it seems that /modules/webserver/manifests/php5.pp only includes the apache module
[17:30:20] <YuviPanda>	 negative24:  https://gerrit.wikimedia.org/r/182840
[17:31:19] <negative24>	 YuviPanda: Looks good
[17:31:44] <YuviPanda>	 negative24: merged
[17:32:07] <negative24>	 So if I run a puppet run it should be there?
[17:32:19] <YuviPanda>	 negative24: yup
[17:32:40] <negative24>	 Ok. thanks
[17:32:56] <Technical_13>	 YuviPanda: Wouldn't that be negative24 affirmative?
[17:33:14] <YuviPanda>	 Technical_13: hmm?
[17:33:26] <Technical_13>	 nevermind... bad joke.  carry on...
[17:33:34] <negative24>	 Oh the references. Thanks Technical_13
[17:33:50] <YuviPanda>	 Technical_13: oh, hah :D
[17:34:20] <negative24>	 YuviPanda: Puppet run worked as advertised
[17:34:49] <YuviPanda>	 yay :)
[17:59:09] <YuviPanda>	 Coren: when you ran maintain-replicas did you have it run only for ipblocks?
[17:59:24] <YuviPanda>	 Coren: can you run it for ipblocks_ipindex too?
[17:59:30] <YuviPanda>	 my earlier fix (replacing 0 with NULL) hasn’t been applied yet
[18:19:53] <Coren>	 Yes, and yes.
[18:20:32] <Coren>	 Running maintain-replicas for all tables takes forever and a half, especially because of the common locks on pagelinks and revision.
[18:20:45] <Coren>	 Running in ipblocks_ipindex now
[18:20:55] <Coren>	 YuviPanda: ^^
[18:21:08] <YuviPanda>	 Coren: ool :)
[18:21:09] <Coren>	 FYI, '-u tablename' when you want to do that
[18:21:09] <YuviPanda>	 *cool
[19:09:52] <wikibugs>	 3Wikimedia-Labs-Infrastructure: make Debian Jessie image for labs - https://phabricator.wikimedia.org/T75592#955819 (10Andrew) This is now mostly working -- I've committed some patches to bootstrap-vz and built a custom package.  Marc is currently working on lvm vs. systemd, it seems that many Jessie users are r...
[19:52:19] <grrrit-wm>	 (03PS1) 10Mattflaschen: Add MoodBar and WikiLove to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/182869 
[20:06:41] <Coren>	 YuviPanda: Btw, ipblocks_ipindex should be done a long time ago.  All good?
[20:07:07] <YuviPanda>	 Coren: yup, yup. 
[20:07:24] <YuviPanda>	 Coren: some discrepancies in the reports, though. some page views have a column named ‘page_no_title_convert’ and some do not...
[20:07:41] <YuviPanda>	 some have rc_cur_time missing
[20:07:45] <Coren>	 That reflects the crazy in the production DBs.
[20:08:47] <Coren>	 Cleaning those up is a sisyphean task that would require at least two Seans.  In addition to the one we do have.
[20:09:13] <YuviPanda>	 Coren: yeah
[20:09:17] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Continuous-Integration: CI labs instances can't start on reboot: tmpfs: Bad value 'jenkins-deploy' for mount option 'uid' - https://phabricator.wikimedia.org/T76250#955973 (10hashar) >>! In T76250#955365, @Krinkle wrote: > By the way, do we have a ticket to track LDAP not being av...
[20:09:25] <YuviPanda>	 Coren: however, we can migrate the views to be fully per-column whitelisted
[20:09:42] <YuviPanda>	 Coren: no ‘full views'
[20:09:58] <Coren>	 That's an option; and you're restrict this to the current schema then?
[20:10:20] <Coren>	 That means some historical data becomes unavailable.  I don't know that anyone uses it, but it's still a breaking change.
[20:10:32] <YuviPanda>	 Coren: well, whitelsted.yaml and greylisted.yaml in labsdb-auditor. When new columns need to be added, they get added to the schema.
[20:10:49] <Coren>	 It's a reasonable approach too.
[20:11:24] <YuviPanda>	 Coren: we can perhaps find some way of sampling queries run against dbs to see if anyone uses them. I highly suspect they don’t - page_no_title_convert was removed in 2006 or something, I think
[20:11:58] <YuviPanda>	 Coren: that will also give us the opportunity to rewrite maintain-replicas in python, using the same data from labsdb-auditor
[20:12:13] <Coren>	 That's a bug, not a feature.  But yeah.
[20:12:37] <YuviPanda>	 although, having two implementations means bugs in one might be caught by the other
[20:12:38] <YuviPanda>	 so I’m not so sure
[20:12:51] <Coren>	 You know, that's actually a good point.
[20:13:18] <YuviPanda>	 but sharing the data perhaps is a good idea.
[20:13:19] <Coren>	 That introduces a consistency check.
[20:13:33] <YuviPanda>	 yeah
[20:13:43] <Coren>	 Well no, it's the independently maintained data that provides the consistency.  :-)
[20:14:07] <YuviPanda>	 yeah
[20:14:10] <YuviPanda>	 so I’m not so sure
[20:14:13] <Coren>	 But fwiw, I agree with the principle of "views are consistent with extactly the current schema" with historical data hidden.
[20:14:27] <Coren>	 There can be no surprises lurking then.
[20:14:39] <YuviPanda>	 Coren: by current schema you mean ‘current schema running in prod’, right?
[20:14:44] <YuviPanda>	 not ‘current schema in labsdb'
[20:14:46] <YuviPanda>	 because latter is crazy
[20:14:54] <Coren>	 I mean "current schema as it's supposed to be in prod"
[20:15:15] <Coren>	 Prod has bazillions of abandonned tables and leftover columns.
[20:15:26] <YuviPanda>	 oh yeah
[20:15:28] <YuviPanda>	 that’s what I meant too
[20:15:30] <Coren>	 Also, not the same ones depending on projects.
[20:15:31] <YuviPanda>	 ‘should be'
[20:15:51] <YuviPanda>	 Coren: yeah, the report contains list of extra tables / columns
[20:15:56] <YuviPanda>	 it’s pretty fucking big
[20:19:37] <hashar>	 Coren: if you have some spare cycles I have a couple puppet patches for you :-]
[20:20:10] <Coren>	 hashar: I need to go grab cafeine, but I'll take a look at them in a bit?
[20:20:21] <hashar>	 sure thing
[20:20:26] <hashar>	 coffee >>> all !
[20:30:46] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: move ops-* to operations-feed channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182880 
[20:32:29] <grrrit-wm>	 (03CR) 10RobH: [C: 031] "dear <insert your higher belief stucture here> yes!" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182880 (owner: 10Merlijn van Deen)
[20:33:46] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] move ops-* to operations-feed channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182880 (owner: 10Merlijn van Deen)
[20:34:00] <grrrit-wm>	 (03CR) 10Hoo man: "I would like to keep it, but whatever" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182880 (owner: 10Merlijn van Deen)
[20:34:04] <grrrit-wm>	 (03Merged) 10jenkins-bot: move ops-* to operations-feed channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182880 (owner: 10Merlijn van Deen)
[20:34:14] <wikibugs>	 3Wikimedia-Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#956034 (10Nemo_bis) The simplest solution here is probably to let the cluster access some wmflabs.org domain, which can then also be added to whi...
[20:37:51] <valhallasw`cloud>	 YuviPanda: I have been wondering whether we could do something about the highlights, though, as many people use the same name on IRC
[20:38:01] <valhallasw`cloud>	 and highlights on your own bugs are basically useless
[20:38:28] <YuviPanda>	 valhallasw`cloud: hmm, true. 
[20:38:33] <YuviPanda>	 valhallasw`cloud: use another name field?
[20:39:13] <valhallasw`cloud>	 yeah, or split the name and add a zero-width space in there?
[20:39:14] <Coren>	 hashar: Tea for me at this hour.  :-)  Point me at 'em
[20:39:33] <hashar>	 Coren: so before xmas break, I had an issue with Jenkins instance stalling on boot
[20:39:47] <hashar>	 Coren: because we attempted to mount a tmpfs that had owner jenkins-deploy,  a user that is only in ldap
[20:39:59] <Coren>	 Too early, then
[20:40:04] <hashar>	 Coren: and since ldap is not available during mountall phase, we ended up with a dead locked instance
[20:40:12] <hashar>	 so instead of figuring out how to fix the boot sequence
[20:40:16] <YuviPanda>	 valhallasw`cloud: I dunno if clients will be smart enough for zero-width
[20:40:28] <valhallasw`cloud>	 YuviPanda: they won't. That's the entire idea ;-)
[20:40:38] <hashar>	 Coren: I changed the tmpfs ownership to root:root and mode 1777 :-]  so it is world writable and mountable early on
[20:40:52] <valhallasw`cloud>	 if you client is smart enough to highlight you on Yuvi<zero width space>Panda, it can also ignore wikibugs in a single channel
[20:40:53] <YuviPanda>	 valhallasw`cloud: hah! hmm, that’s worth trying out, sure
[20:41:21] <valhallasw`cloud>	 legoktm, what do you think? I'm not sure where to insert the nbsp though :-p
[20:41:24] <valhallasw`cloud>	 er, zwsp
[20:41:27] <hashar>	 Coren: the puppet patch is https://gerrit.wikimedia.org/r/#/c/173511/  (deployed on CI labs instance already )
[20:41:34] <YuviPanda>	 valhallasw`cloud: EVERYWHERE ;)
[20:41:37] <YuviPanda>	 one after every char
[20:41:41] <valhallasw`cloud>	 YuviPanda: but max line length!
[20:41:41] <YuviPanda>	 or just insert one midway through
[20:41:43] <YuviPanda>	 PICK A SPOT!
[20:41:45] <YuviPanda>	 :D
[20:42:07] <hashar>	 I am only notified on "hashar", not on "antoine musso" :-D
[20:42:39] <valhallasw`cloud>	 using the 'aka' field is also an option. You don't like highlights? choose a better 'aka' name!
[20:42:45] <valhallasw`cloud>	 I like that better, actually.
[20:44:39] <legoktm>	 valhallasw`cloud: I actually like getting highlighted for my bug changes.
[20:44:49] <valhallasw`cloud>	 legoktm: but you just did them yourself :o
[20:44:56] <legoktm>	 https://xkcd.com/1172/ !!
[20:45:01] <hashar>	 Coren: yeah the sticky on the tmpfs comes from /tmp/  I copy pasted :D
[20:45:07] <valhallasw`cloud>	 now I'm fascinated by what your workflow is :P
[20:45:23] <valhallasw`cloud>	 legoktm: also, if we use the AKA field, you can just add your irc nick there! ;-D
[20:46:04] <legoktm>	 ah yes, legoktm aka legoktm
[20:46:22] <hashar>	 Coren: and the next patch actually enable that tmpfs on CI labs instances  with 512MB of ram : https://gerrit.wikimedia.org/r/#/c/173512/   deployed already
[20:46:43] <valhallasw`cloud>	 legoktm: we can ask the phab team to add a 'wikibugs should call me...' field!
[20:47:05] <valhallasw`cloud>	 legoktm: still, are you actually using those highlights?
[20:47:15] <YuviPanda>	 valhallasw`cloud: I like getting highlighted on my gerrit changes, so I can click them to open in browser
[20:47:17] <legoktm>	 heh
[20:47:28] <valhallasw`cloud>	 YuviPanda: yeah, that use case I get.
[20:47:30] <legoktm>	 yes, I do use those highlights
[20:47:48] <valhallasw`cloud>	 YuviPanda: although the url is right there when you push! (not for PS2 and higher, though, I think :/)
[20:47:49] <legoktm>	 mainly it makes it easier to find things in scrollback
[20:48:02] <valhallasw`cloud>	 "x happened after I posted y"?
[20:48:06] <YuviPanda>	 valhallasw`cloud: yeah, but I’ve to command click
[20:48:06] <YuviPanda>	 there
[20:48:07] <YuviPanda>	 instead of, uh, command tabbing + clicking
[20:52:07] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: c45fa33e94f5a34fa7618eaad1669104ace2a342 Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2
[20:52:36] <labs-morebots>	 Logged the message, Master
[20:54:04] <renoirb>	 Is there a way to add content on the /Special:UserLogin?type=signup through Transclusion?  I’ve been searching around and cannot find which to use.   MediaWiki:UserLogin, MediaWiki:CreateAccount ? 
[20:55:21] <YuviPanda>	 renoirb: you probably want to ask in #mediawiki or #wikimedia-dev
[20:55:37] <bd808>	 renoirb: Throw a "&uselang=qqx" on the end of the URL to see the messages it uses in each area
[20:55:39] <renoirb>	 I did YuviPanda  :) its in a torrent of gerrit notifications :)
[20:55:59] <renoirb>	 oh! that’s what legoktm told me but didnt thought it was adressed to me 
[20:56:11] * bd808 sees "signupstart"
[20:56:38] <YuviPanda>	 heh :)
[20:56:38] <renoirb>	 oh, its part of the messages system thing.
[20:56:49] <renoirb>	 I dont remember how to access it, i’ll take a note this time.
[20:57:14] <renoirb>	 I guess its in Special:AllMessages
[20:57:18] <renoirb>	 with the matching key?
[21:02:11] <bd808>	 renoirb: bascially once you find a key you can create a page at "MediaWiki:Key" to replace the default translation for that string. (eg MediaWiki:Signupstart to put stuff before the new account creation form)
[21:02:49] <renoirb>	 Cool! I’ll keep &uselang=qqx at hand!!
[21:03:33] <YuviPanda>	 legoktm: valhallasw`cloud my parser! :) https://gerrit.wikimedia.org/r/#/c/182848/
[21:03:37] <YuviPanda>	 well, it’s fairly trivial, but still
[21:11:01] <valhallasw`cloud>	 YuviPanda: nice
[21:11:19] <valhallasw`cloud>	 doesn't pyparsing just have a built-in SQL parser? *runs*
[21:12:00] <YuviPanda>	 valhallasw`cloud: http://pyparsing.wikispaces.com/file/view/select_parser.py/158651233/select_parser.py
[21:12:06] <YuviPanda>	 valhallasw`cloud: but that’s very complicated for my needs
[21:12:18] <valhallasw`cloud>	 YuviPanda: also well-tested (maybe not, though(
[21:12:20] <YuviPanda>	 valhallasw`cloud: and doesn’t handle things like ` quoting properly
[21:12:25] <valhallasw`cloud>	 I see
[21:12:52] <YuviPanda>	 valhallasw`cloud: it handles a lot more cases correctly than the thing I wrote
[21:13:01] <YuviPanda>	 valhallasw`cloud: but I only need it to handle specific cases
[21:28:07] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Add #wikidata umbrella to #wikidata [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182888 
[21:29:01] <grrrit-wm>	 (03CR) 10Hoo man: [C: 031] Add #wikidata umbrella to #wikidata [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182888 (owner: 10Merlijn van Deen)
[21:31:19] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Add #wikidata umbrella to #wikidata [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182888 (owner: 10Merlijn van Deen)
[21:31:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add #wikidata umbrella to #wikidata [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/182888 (owner: 10Merlijn van Deen)
[21:32:04] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 9003536427ce097a62c3a8cec310f7ca4f0edab0 Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2
[21:32:10] <labs-morebots>	 Logged the message, Master
[22:03:52] <wikibugs>	 3Tool-Labs, Labs-Team: Document labsdb replication set up - https://phabricator.wikimedia.org/T85868#956362 (10yuvipanda) 3NEW
[23:56:43] <wikibugs>	 3Beta-Cluster, operations, Labs-Team: Backport new salt-syndic packages - https://phabricator.wikimedia.org/T85442#956706 (10greg) p:5Triage>3Normal