[00:00:12] <bd808>	 it's not hooked up to prod centralauth
[00:00:15] <quiddity>	 I use a unique password, but my usual username.
[00:00:25] <bd808>	 *nod*
[00:01:12] <bd808>	 quiddity: works for me logged in. Do you have any weird gadgets or anything?
[00:01:34] <quiddity>	 I was experimenting with the mobile-frontend-experimental build, at the time...
[00:02:52] <bd808>	 quiddity: may be a real bug. #wikimedia-releng would be a good place to show for help finding log messages
[00:03:05] <bd808>	 s/show/shop/
[00:03:12] <quiddity>	 will do, thanks.
[00:19:21] <quiddity>	 (all fixed. jdlrobson reset my preferences, which fixed it)
[03:18:10] <YuviPanda>	 legoktm: is redis still dead?
[03:20:43] <legoktm>	 YuviPanda: I haven't tried restarting wikibugs or grrrit-wm
[03:55:10] <YuviPanda>	 legoktm: hmm
[03:56:32] <YuviPanda>	 !log tools restarted redis server, it had OOM-killed
[03:56:41] <labs-morebots>	 Logged the message, Master
[03:57:10] <mutante>	 ls
[03:59:09] <^demon|away>	 YuviPanda: Can you shoot something on iridium for me again?
[03:59:10] <^demon|away>	 https://phabricator.wikimedia.org/T92360
[03:59:19] <YuviPanda>	 ^demon|away: sure
[03:59:24] <^demon|away>	 thx
[04:03:28] <YuviPanda>	 ^demon|away: done yo
[04:03:46] <^demon|away>	 thx bro
[04:12:42] <legoktm>	 YuviPanda: do you know about https://phabricator.wikimedia.org/T92313?workflow=create ?
[04:13:03] <YuviPanda>	 legoktm: I’m redoing redis, i’ll brb in a bit
[04:13:28] <shinken-wm>	 PROBLEM - Host tools-redis is DOWN: CRITICAL - Host Unreachable (10.68.17.132)  
[04:14:38] <YuviPanda>	 yeah, I know...
[04:14:47] <YuviPanda>	 !log tools kill tools-redis instance, upgrade to trusty while it is down anyway
[04:14:52] <labs-morebots>	 Logged the message, Master
[04:15:52] <legoktm>	 YuviPanda: oh wat I just started grrrit and wikibugs
[04:15:53] <legoktm>	 lol
[04:17:26] <shinken-wm>	 RECOVERY - Host tools-redis is UP: PING OK - Packet loss = 0%, RTA = 0.87 ms  
[04:28:18] <YuviPanda>	 !log tools tools-redis is back now, as trusty and hopefully slightly more fortified
[04:28:19] <YuviPanda>	 legoktm: ^
[04:28:22] <labs-morebots>	 Logged the message, Master
[04:28:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0]  
[04:31:01] <YuviPanda>	 !log tools.lolrrit-wm restarted grrrit-wm and gerrit-to-redis
[04:31:05] <labs-morebots>	 Logged the message, Master
[04:31:24] <YuviPanda>	 legoktm: this also moves us to a much more recent version of redis...
[04:33:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0]  
[04:36:29] <legoktm>	 YuviPanda: woot
[04:36:57] <legoktm>	 !log tools.wikibugs restarted both wb2-phab and wb2-irc
[04:37:01] <labs-morebots>	 Logged the message, Master
[06:17:30] <YuviPanda>	 legoktm: so wikibugs is fine, grrrit-wm is still dead
[07:55:02] <grrrit-wm>	 (03PS1) 10Yuvipanda: Directly interface with gerrit stream-events [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195861 
[07:55:08] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Directly interface with gerrit stream-events [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195861 (owner: 10Yuvipanda)
[07:56:02] <grrrit-wm>	 (03PS2) 10Yuvipanda: Directly interface with gerrit stream-events [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195861 
[08:20:22] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: Puppetize LVM extension for tools redis - https://phabricator.wikimedia.org/T91370#1108662 (10yuvipanda) Done differently in I976f4c29d6730bd563ae6fb7a33c86b6249705d2, has redis data in /srv instead.
[08:20:52] <wikibugs>	 10Tool-Labs, 5Patch-For-Review: Puppetize LVM extension for tools redis - https://phabricator.wikimedia.org/T91370#1108664 (10yuvipanda) 5Open>3Resolved a:3yuvipanda
[10:36:45] <werdna>	 hey YuviPanda, do I have to do anything to get access to beta.wmflabs.org?
[10:37:17] <YuviPanda>	 werdna: hey! I 
[10:37:22] <YuviPanda>	 werdna: I guess you could ask the releng people...
[10:37:44] <YuviPanda>	 werdna: by ‘access’ you mean ssh or?
[10:37:53] <werdna>	 something like that
[10:38:04] <werdna>	 the user research people want me to import something onto en.wikipedia.wmflabs.org
[10:38:08] <YuviPanda>	 ah
[10:38:16] <YuviPanda>	 you… probably should co-ordinate that with RelEng
[10:38:22] <YuviPanda>	 I can add you already, however.
[10:38:34] <werdna>	 I'll ask
[10:38:41] <YuviPanda>	 alright.
[10:38:50] * YuviPanda desists
[12:12:21] <Cyberpower678>	 Can someone tell me how to disable big brother.  I cannot for the life of me get it to stop.
[12:12:50] <Cyberpower678>	 It's going on a wild rampage on xTools.  I think something about big brother got borked.
[12:14:10] <Cyberpower678>	 YuviPanda, 
[12:14:41] <YuviPanda>	 hi Cyberpower678 
[12:14:44] <YuviPanda>	 what do you mean by wild rampage
[12:14:49] <YuviPanda>	 I see only one job on https://tools.wmflabs.org/?status
[12:14:51] <YuviPanda>	 for it
[12:15:34] <Cyberpower678>	 It keeps rebooting xTools.  Recently we've had our mailbox spammed with 100+ emails from big brother.
[12:16:42] <Cyberpower678>	 We want to switch to our own web watcher which is a little more advanced than big brother.
[12:16:52] <Cyberpower678>	 So Itried disabling bigbrother
[12:17:02] <Cyberpower678>	 YuviPanda, ^ but it won't go away.
[12:17:04] <YuviPanda>	 Cyberpower678: oh, hmm. if it isn’t an active issue, I suggest filing a bug and poking Coren when he’s around. 
[12:17:14] <YuviPanda>	 so that sounds like a bug (can’t opt-out of bigbrother)
[12:17:20] <YuviPanda>	 (and a terrible one)
[12:17:27] <YuviPanda>	 Cyberpower678: however, I also thought you were moving xtools to its own VM?
[12:17:40] <Cyberpower678>	 I tried deleting the file, and then simply blanking it, but it just keeps coming back.
[12:18:00] <Cyberpower678>	 We are, but we're slow with setting it up.
[12:18:05] <YuviPanda>	 fair enough
[12:18:18] <Cyberpower678>	 We are learning the environment.
[12:18:20] <YuviPanda>	 Cyberpower678: alright, so 1. delete the file, and I’ll restart bigbrother itself. and we’ll see if that helps?
[12:18:32] <Cyberpower678>	 I have no access right now
[12:18:55] <YuviPanda>	 alright. I’ll wait till you do have access :)
[12:18:58] <Cyberpower678>	 Could you do it for me for xtools, xtools-ec, and xtools-articleinfo
[12:19:03] <Cyberpower678>	 Ok.
[12:19:06] <YuviPanda>	 sorry, neck deep in some betalabs stuff
[13:35:23] <wikibugs>	 10Tool-Labs: Fix and clean up generation of ssh_known_keys - https://phabricator.wikimedia.org/T92379#1109287 (10scfc) 3NEW a:3scfc
[14:14:35] <wikibugs>	 10Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1109425 (10coren) a:3coren
[14:24:20] <wikibugs>	 10Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1109452 (10coren) The one thing I am missing to implement this at this time is either (a) the whitelist of properties to collate or (b) the method by which that whitelist is generated.
[14:25:30] <wikibugs>	 10Tool-Labs: Audit redis usage on toollabs - https://phabricator.wikimedia.org/T91979#1109463 (10coren) a:3coren
[14:25:38] <wikibugs>	 10Tool-Labs: Audit redis usage on toollabs - https://phabricator.wikimedia.org/T91979#1100333 (10coren) p:5Triage>3High
[14:28:56] <Coren>	 YuviPanda: After data collection, the biggest source of redis use is now known.  It's not clear, because of limited data, whether it's also the bigger user of keys but it's likely (takes roughly 50% of usage on its own)
[14:29:36] <YuviPanda>	 Coren: ah. who is it?
[14:29:43] <Coren>	 YuviPanda: anomiebot
[14:29:47] <YuviPanda>	 Coren: oooh, interesting.
[14:29:59] <YuviPanda>	 Coren: btw, I re-imaged the box to be trusty + set limit at 12G, so OOM killer won’t get to redis
[14:30:06] * YuviPanda pokes anomie
[14:30:11] <Coren>	 That'll help.
[14:30:11] <YuviPanda>	 Coren: do they have ttl set?
[14:30:38] * anomie didn't think anomiebot was that big a user of redis
[14:30:55] <Coren>	 YuviPanda: Not that I can see.
[14:31:00] <YuviPanda>	 Coren: how are you measuring this, btw?
[14:31:02] <Coren>	 anomie: There might be a bug?
[14:31:04] <YuviPanda>	 I also switched from aof to rdb
[14:31:21] <Coren>	 YuviPanda: Monitoring wire traffic.  Turns out to have been the most reliable way.  :-)
[14:31:26] <YuviPanda>	 ah
[14:31:35] <anomie>	 As far as keys, all AnomieBOT's keys should be prefixed with "AnomieBOT:" + a random-looking string.
[14:31:48] <Coren>	 anomie: We can't actually /list/ keys.  :-)
[14:32:14] <anomie>	 Coren: There's no special admin access to list them?
[14:32:31] <Coren>	 anomie: But the biggest simple change you could make is set a TTL on the keys if they have limited lifetime.
[14:33:35] <YuviPanda>	 (I emailed a reminder to labs-l)
[14:53:04] <anomie>	 Coren, YuviPanda: I just checked AnomieBOT's code, and it turns out a timeout already is being set on just about everything where a timeout would make sense. Mostly 86400 seconds, 7200s for redirect info (i.e. title=>targets and target=>title), and interwiki/namespace prefix lists for 604800s (7 days). The only thing that's not already setting a timeout that might deserve one is the cache of what aliases there are for "#REDIRECT".
[14:53:46] <YuviPanda>	 anomie: hmm, fair enough. 
[14:54:06] <YuviPanda>	 Coren: we should look at key sizes (via rdb parsing, I think?) since our problem isn’t throughput but storage...
[14:54:09] <YuviPanda>	 well, ‘size’ rather.
[14:54:25] <anomie>	 The other things that don't are the keys behind https://tools.wmflabs.org/anomiebot
[14:54:27] <YuviPanda>	 anomie: how many places do you have where it doesn’t make sense to set a timeout?
[14:54:35] <anomie>	 YuviPanda: Heh, I just answered that
[14:54:39] <YuviPanda>	 heh
[14:55:15] <YuviPanda>	 anomie: nice :) have you switched to trusty already, btw? :)
[14:55:37] <anomie>	 YuviPanda: For web stuff, yes. For grid jobs, probably not.
[15:01:01] <YuviPanda>	 anomie: hmm, ok. it doesn’t make as much of a difference in grid jobs yet. Maybe in several more months...
[15:02:07] <anomie>	 I'm not specifying any release, and I don't anticipate anything breaking since it hasn't when I run stuff locally for testing.
[15:03:56] <YuviPanda>	 anomie: cool
[15:03:59] <YuviPanda>	 anomie: thanks :)
[15:06:56] <YuviPanda>	 legoktm: https://gerrit.wikimedia.org/r/#/c/195861/ gets rid of redis dependency for grrrit-wm :)
[15:08:39] <YuviPanda>	 legoktm: I’m going to merge now, and hopefully it won’t break
[15:08:52] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Directly interface with gerrit stream-events [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195861 (owner: 10Yuvipanda)
[15:12:26] <ircnotifier>	 !log tools.lolrrit-wm yuvipanda: Deployed 163bac22f003c83443c69d03efed65d8c358ab8c Directly interface with gerrit stream-events
[15:12:30] <labs-morebots>	 Logged the message, Master
[15:12:42] <YuviPanda>	 grrrit-wm: why haven’t you restarted?!
[15:12:43] <YuviPanda>	 aaah
[15:12:44] <YuviPanda>	 :D
[15:12:45] <YuviPanda>	 there we go
[15:14:47] <legoktm>	 YuviPanda: ooh!
[15:15:16] <YuviPanda>	 legoktm: aargah, it isn’t actually working...
[15:15:19] <YuviPanda>	 worked fine when I was testing...
[15:15:20] <YuviPanda>	 oh
[15:15:20] <YuviPanda>	 wait
[15:15:22] <YuviPanda>	 config change
[15:20:21] <James_F>	 YuviPanda: Having fun?
[15:20:27] <YuviPanda>	 James_F: yup
[15:20:40] <YuviPanda>	  James_F however, once it comes back up, it should be more reliable because it has a *lot* less moving parts
[15:20:47] <James_F>	 Nice.
[15:21:24] <James_F>	 YuviPanda: Feel like pushing https://gerrit.wikimedia.org/r/#/c/195569/ too whilst you're there?
[15:21:40] <YuviPanda>	 James_F: I shall, after I bring it back to life
[15:21:45] <James_F>	 Ta.
[15:38:04] <grrrit-wm>	 (03CR) 10Yuvipanda: "It was precise's fault, wasn't it?" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195861 (owner: 10Yuvipanda)
[15:38:12] <YuviPanda>	 of course it was :D
[15:39:11] <grrrit-wm>	 (03PS1) 10Yuvipanda: Run on trusty nodes [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195921 
[15:39:55] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Run on trusty nodes [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195921 (owner: 10Yuvipanda)
[15:39:58] <grrrit-wm>	 (03Merged) 10jenkins-bot: Run on trusty nodes [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195921 (owner: 10Yuvipanda)
[15:40:11] <grrrit-wm>	 (03PS3) 10Yuvipanda: Add some obvious repos to -releng (and scap to -operations too) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195569 (owner: 10Jforrester)
[15:42:54] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Add some obvious repos to -releng (and scap to -operations too) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195569 (owner: 10Jforrester)
[15:42:58] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add some obvious repos to -releng (and scap to -operations too) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/195569 (owner: 10Jforrester)
[15:43:08] <YuviPanda>	 James_F: ^ done
[15:44:09] <ircnotifier>	 !log tools.lolrrit-wm yuvipanda: Deployed 4d0fbb28c9ddf9dec60c03e1a1084fc3ad42cd60 Add some obvious repos to -releng (and scap to -operations too)
[15:44:13] <labs-morebots>	 Logged the message, Master
[16:05:51] <James_F>	 YuviPanda: Thanks!
[16:48:25] <wikibugs>	 10Tool-Labs: Provide a link to source code of tools.wmflabs.org - https://phabricator.wikimedia.org/T92394#1109831 (10Petrb) 3NEW
[16:49:25] <wikibugs>	 10Tool-Labs: Provide a link to source code of tools.wmflabs.org - https://phabricator.wikimedia.org/T92394#1109839 (10Petrb)
[17:02:42] <CP678|iPhone>	 YuviPanda: BugBro is doing it again.  I'm getting a flood of restart emails.
[17:02:59] <YuviPanda>	 Coren: ^
[17:03:17] <CP678|iPhone>	 And it should technically be off.
[17:03:18] <YuviPanda>	 I still see only one set of jobs though
[17:03:39] <CP678|iPhone>	 I blanked the .bigbrotherrc files
[17:06:40] <CP678|iPhone>	 It still shouldn't be fidgeting with xTools.
[17:07:17] <Coren>	 Wait, flood of restart email or flood of "I couldn't start your tool" email?  Because it shouldn't even be /trying/ more than three times in a 24h window
[17:07:29] <YuviPanda>	 Coren: he’s trying to opt out of bigbrother and it wont’ let him
[17:08:37] <Coren>	 Which tool is that?
[17:10:33] <CP678|iPhone>	 Both and what  YuviPanda said.  For xTools, xTools-ec, and xTools-articleinfo
[17:10:56] <CP678|iPhone>	 I'm also getting a failed to start services messages.
[17:12:00] <CP678|iPhone>	 I have to go for now.
[17:13:23] <CP678|iPhone>	 Coren: ^^
[17:13:41] <Coren>	 CP678|iPhone: I'm gonna be looking into it.  It's clearly a bug.
[17:14:19] <CP678|iPhone>	 :-)
[17:14:23] <CP678|iPhone>	 Cya
[17:23:38] <Niharika>	 YuviPanda: I was wondering if you'd like to be a GSoC mentor for the upcoming round. 
[17:24:20] <YuviPanda>	 Niharika: probably, assuming I can find the right project...
[17:24:27] <YuviPanda>	 Niharika: actually, I’ll be moving continents at that time, so maybe not?
[17:25:02] <Niharika>	 YuviPanda: Ah. Okay. I am mentor-hunting for https://phabricator.wikimedia.org/T5525 Any ideas who'd be interested? 
[17:25:56] <YuviPanda>	 Niharika: ah, probably not me. legoktm or someone in mediawiki-core maybe. It also might be too big a project for GSoC
[17:26:22] <Niharika>	 YuviPanda: By the way, congratulations! :D
[17:26:28] <YuviPanda>	 Niharika: :) ty
[17:26:34] <legoktm>	 yeah, that might be a little large for a GSoC project...
[17:26:52] <legoktm>	 maybe if you scrap the UI part of it and just do backend implementation
[17:27:59] <Niharika>	 legoktm: Okay. We could do that. There's interest in pursuing this, AFAICS. 
[17:28:22] <legoktm>	 Niharika: there's definitely interest, but it's a *hard* problem.
[17:29:38] <Niharika>	 legoktm: Right. 
[18:00:45] <a930913>	 YuviPanda: The redis pubsub stuff doesn't fill the server does it?
[18:01:00] <YuviPanda>	 a930913: pubsub shouldn’t, no.
[18:01:11] <YuviPanda>	 a930913: although me and legoktm are thinking of setting up a tools-redis-pubsub just for pubsub
[18:01:14] <YuviPanda>	 (and queues)
[18:02:25] <a930913>	 YuviPanda: But migration :p Is there such a specification difference?
[18:02:27] <wikibugs>	 10Tool-Labs-tools-Other: Non technical: "Database reports" - status query - https://phabricator.wikimedia.org/T92353#1110265 (10Aklapper)
[18:02:57] <YuviPanda>	 a930913: no. mostly we’ll disallow normal key activities on the pubsub / queue redis, so it won’t die because of overloaded caches :D
[18:03:06] <YuviPanda>	 but that might not be necessary. let’s see how the latest fortifications hold up
[18:03:57] <a930913>	 YuviPanda: redis-stable and redis-experimental? :p
[18:04:06] <YuviPanda>	 nope :P
[18:04:14] <YuviPanda>	 redis-cache and redis-queue maybe
[18:04:49] <a930913>	 :D
[18:27:14] <wikibugs>	 10Tool-Labs-tools-Other: Non technical: "Database reports" - status query - https://phabricator.wikimedia.org/T92353#1110357 (10Legoktm) https://github.com/mzmcbride/database-reports/issues/13 and https://github.com/mzmcbride/database-reports/issues/14 covers why reports aren't running currently.
[18:45:24] <Weiroutsi>	 Tool is down: https://tools.wmflabs.org/kmlexport/?project=de&linksfrom=1&article=Cham&redir=bing
[18:47:07] <YuviPanda>	 !log deployment-prep created deployment-mediawiki03
[18:47:12] <labs-morebots>	 Logged the message, Master
[18:47:50] <YuviPanda>	 Weiroutsi: I just rebooted it
[18:56:20] <Weiroutsi>	 @YuviPanda: Thank you!
[18:56:56] <YuviPanda>	 yw :)
[19:28:07] <wikibugs>	 10Tool-Labs: Provide a link to source code of tools.wmflabs.org - https://phabricator.wikimedia.org/T92394#1110600 (10scfc)
[19:28:08] <wikibugs>	 10Tool-Labs: Provide source/repository link on https://tools.wmflabs.org - https://phabricator.wikimedia.org/T86431#1110601 (10scfc)
[19:39:54] <wikibugs>	 6Labs, 10Wikimedia-Labs-wikitech-interface: Use a Puppet ENC to define which classes are included in which nodes (in Labs) - https://phabricator.wikimedia.org/T85279#1110728 (10yuvipanda) If we make it query LDAP, we can actually have this be an addition to the LDAP terminus for labs. Maybe even parameterize i...
[19:40:16] <wikibugs>	 6Labs, 10Wikimedia-Labs-wikitech-interface: Use a Puppet ENC to define which classes are included in which nodes (in Labs) - https://phabricator.wikimedia.org/T85279#1110729 (10yuvipanda) a:3yuvipanda
[20:05:45] <wikibugs>	 10Tool-Labs-tools-Other: Non technical: "Database reports" - status query - https://phabricator.wikimedia.org/T92353#1110813 (10Haruth) Thanks for the info.   Will leave well alone on the basis that things are happening behind the scenes :)
[20:57:23] <wikibugs>	 6Labs, 7Puppet: Puppet Trebuchet provider compares refname with commit sha1 and does NOT refresh the git repo! - https://phabricator.wikimedia.org/T77002#1111012 (10chasemp) p:5High>3Normal
[20:58:06] <tgr>	 did the process to create instances change recently?
[20:58:18] <tgr>	 I see no admin links on Special:NovaInstance
[20:58:21] <grrrit-wm>	 (03PS1) 10Yuvipanda: Use ssh::userkey for root as well [labs/private] - 10https://gerrit.wikimedia.org/r/196019 
[20:58:55] <YuviPanda>	 tgr: nope. if you are seeing blanks, try logging out and back in
[20:59:54] <tgr>	 YuviPanda: that worked, thanks
[21:00:00] <YuviPanda>	 tgr: yw!
[21:00:13] * YuviPanda goes back to his IT Crowd-esque ioffice
[21:08:35] <multichill>	 YuviPanda: Any idea why queries would be much much slower?
[21:08:55] <multichill>	 https://tools.wmflabs.org/multichill/queries/wikidata/noclaims_nlwiki.txt took 43m48.979s, it's usually a comple of minutes
[21:09:08] <YuviPanda>	 multichill: ugh, no idea...
[21:09:25] <multichill>	 Do we keep graphs of the load of the db servers?
[21:09:31] <YuviPanda>	 yeah...
[21:09:38] <YuviPanda>	 looking at them now
[21:10:11] <multichill>	 I ran this yesterday evening
[21:11:17] <YuviPanda>	 multichill: hmm, so it’s 3AM and my brain isn’t working :(
[21:11:26] <YuviPanda>	 can you file a bug? maybe Coren can take a look (cc springle as well)
[21:11:35] <YuviPanda>	 and I’ll poke around tomorrow if nobody else gets to it first
[21:11:36] <YuviPanda>	 sorry
[21:11:49] <multichill>	 Just wondering
[21:12:12] <multichill>	 I run that query every once in a while, I'll see how long it takes next time
[21:12:14] <YuviPanda>	 multichill: the graphs are public at ganglia.wikimedia.org
[21:12:15] <YuviPanda>	 heh
[21:12:20] <multichill>	 Right
[21:16:19] <multichill>	 Take it easy YuviPanda
[21:17:18] <YuviPanda>	 I should :)
[22:01:02] * Coren reads scrollback
[22:01:09] <Coren>	 (Out to dinner, sorry)
[22:15:41] <tgr>	 are nfs failures in labs puppet runs normal?
[22:15:43] <tgr>	 "Execution of '/bin/mount /data/project' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/multimedia/project failed, reason given by server: No such file or directory"
[22:15:54] <tgr>	 or is that project misconfigured somehow?
[22:17:37] <JohnFLewis>	 tgr: I believe mounting can fail on the first run. A puppet re-run (or reboot) should fix it iirc
[22:18:05] <tgr>	 this happens on every run (did not try rebooting)
[22:18:14] <tgr>	 not really a problem, just wondering
[22:18:32] <JohnFLewis>	 reboot I think?
[22:29:33] <Coren>	 There are two possibiilties:  Either your instance was booted faster than NFS server was ready - can happen on new instances - or you forgot to actually turn 'project storage' on in the project settings.
[22:29:48] <Coren>	 But the latter, I think, was changed to on by default some time ago.
[22:38:06] <tgr>	 must have been a timing issue
[22:38:11] <tgr>	 rebooting helped, thanks