[00:13:42] <YuviPanda|brb>	 andrewbogott: nope
[00:14:02] <YuviPanda|brb>	 andrewbogott: people are using it
[00:14:22] <andrewbogott>	 YuviPanda|brb: ok
[00:15:14] <YuviPanda|brb>	 Ty
[07:19:50] <jongleur>	 good morning. anybody who can help with a few questions regarding openstreetmap-using projects on labs?
[10:31:44] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Jenkins jobs updating beta cluster should whine - 10https://bugzilla.wikimedia.org/69628 (10Antoine "hashar" Musso) 5NEW>3ASSI p:5Unprio>3Normal s:5normal>3enhanc a:3Antoine "hashar" Musso
[13:13:49] <YuviPanda>	 Coren: ping when around? I've tested the graphite changes on precise as well now :)
[13:15:57] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Jenkins jobs updating beta cluster should whine - 10https://bugzilla.wikimedia.org/69628#c3 (10Antoine "hashar" Musso) 5PATC>3RESO/FIX [12:37:26] <wmf-insecte>     Project beta-scap-eqiad build #17860: FIXED in 8 min 22 sec: https://integration.wikimedia.org/ci/...
[13:27:25] <Coren>	 YuviPanda: Around, but a bit busy.
[13:29:03] <YuviPanda>	 Coren: alright, ping when you've a minute or two to merge :) 
[13:34:57] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708#c3 (10Antoine "hashar" Musso) Here are the hhvm versions deployed on the two beta cluster instances:   deployment-mediawiki01:~$ hhvm --version HipHop VM 3.3.0-dev...
[13:35:12] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): wikidata beta (item pages, etc.) inaccessible with 503 errors - 10https://bugzilla.wikimedia.org/69708 (10Antoine "hashar" Musso)
[13:51:08] <valhallasw`cloud>	 Coren: or a smart spammer that reads mailing list archives :-p
[13:51:31] <Coren>	 valhallasw`cloud: That email stinks of automation.
[13:52:13] <valhallasw`cloud>	 spambot that reads mailing list archives*
[13:53:00] <valhallasw`cloud>	 scrape two mail addresses from the same archive page, send email from one to the other, ???, PROFIT
[13:54:02] <hashar_>	 tonythomas: the beta cluster bastion has some error :-}
[13:54:16] <tonythomas>	 hashar_: true. its not updating stuff from gerrit 
[13:55:23] <tonythomas>	 hashar_: and one more thing regarding labs --> I could generate an email with VERP return-path using BounceHandler extension alright some 17 hourse before - but right now I see that there is some php error in http://en.wikipedia.beta.wmflabs.org/wiki/Special:EmailUser 
[13:55:34] <hashar_>	 !log deployment-prep On bastion, deleting /usr/local/apache/common-local  and symlink it to /srv/common-local
[13:55:37] <labs-morebots>	 Logged the message, Master
[13:56:13] <tonythomas>	 hashar_: something broke that in btw ?
[13:57:24] <hashar_>	 tonythomas: the bastion is just used to fetch code from gerrit
[13:57:31] <hashar_>	 the content is then copied to /srv/scap-stage-dir
[13:57:40] <hashar_>	 and rsynced from there  to the other instances
[13:57:45] <hashar_>	 seems all are up to date 
[13:58:30] <tonythomas>	 hashar_: ok. that looks good. and any idea what broke that Special:EmailUser ?
[14:00:09] <hashar>	 !log deployment-prep On bastion reverted the symlink on bastion and manually created directory /usr/local/apache/common-local
[14:00:11] <labs-morebots>	 Logged the message, Master
[14:17:23] <hashar>	 !log deployment-prep huge rsync in progress on bastion
[14:17:26] <labs-morebots>	 Logged the message, Master
[14:17:30] <YuviPanda>	 andrewbogott: hey! around?
[14:17:49] <andrewbogott>	 YuviPanda: just eating breakfast, back in a few
[14:17:57] <YuviPanda>	 andrewbogott: ah cool
[14:26:39] <andrewbogott>	 YuviPanda: what's up?
[14:26:57] <YuviPanda>	 andrewbogott: merge https://gerrit.wikimedia.org/r/#/c/154712/4?
[14:27:18] * andrewbogott reads
[14:27:56] <YuviPanda>	 andrewbogott: also unsure how to do https://gerrit.wikimedia.org/r/#/c/154854/, since labmon1001 has data storage in /srv, but the role realm branches on $::realm, which will not be labs for labmon, so it'll put things in /var/lib which will die
[14:28:15] <YuviPanda>	 andrewbogott: am wondering if the easiest solution is to just change mount points, point /srv to /var/lib  and /srv/var to /var
[14:30:17] <andrewbogott>	 What's the misc server that runs production graphite?
[14:30:31] <YuviPanda>	 andrewbogott: tungsten
[14:30:43] <YuviPanda>	 mimicing that one's config would be the best
[14:30:50] <YuviPanda>	 andrewbogott: but I don't know how it is configured, since I don't have access
[14:30:52] <andrewbogott>	 ok, and tungsten and labmon1001 just have their big data volumes mounted in different places?
[14:31:27] <YuviPanda>	 andrewbogott: yeah
[14:31:41] <andrewbogott>	 Seems right to rearrange labmon to copy tungsten then, yeah.
[14:31:44] <YuviPanda>	 yeah
[14:31:58] <YuviPanda>	 andrewbogott: think you can do that? I don't have tungsten access..
[14:32:25] <andrewbogott>	 we're changing labmon1001, not tungsten, right?
[14:32:41] <andrewbogott>	 Does this help?  https://dpaste.de/j59A
[14:33:24] <YuviPanda>	 aaaha
[14:33:30] <YuviPanda>	 it's just mounted on /var/lib/carbon
[14:33:32] <YuviPanda>	 and not /var/lib
[14:33:34] <YuviPanda>	 andrewbogott: it does
[14:34:44] <YuviPanda>	 andrewbogott: so I'll just modify fstab...
[14:36:22] <andrewbogott>	 YuviPanda: yep, I'm sure we have some puppet classes for automating that.
[14:37:10] <YuviPanda>	 andrewbogott: oh? right now there's nothing applied on labmon1001, and Coren did the partitioning manually
[14:37:27] <andrewbogott>	 hm, lemme look.  I'm sure we do it with puppet on labs instances.
[14:37:43] <YuviPanda>	 andrewbogott: right. I've modified fstab, but will wait for you to look before I reboot
[14:38:16] <andrewbogott>	 look at how e.g. /data/project is handled in role/labs.pp
[14:38:22] <andrewbogott>	 seems straightforward.
[14:40:02] <YuviPanda>	 andrewbogott: ah, hmm. so I guess i'll have to have a role::labmon
[14:40:26] <andrewbogott>	 yeah, seems right.
[14:40:37] <andrewbogott>	 I take there isn't already a puppet class for that box?
[14:40:40] <andrewbogott>	 *role
[14:42:06] <Coren>	 I'd realy really rather we *fix* the /var/lib thing to point to /srv than vice-versa.
[14:42:21] <Coren>	 Yes, proper use of /srv is one of my battle horses.  :-)
[14:42:44] <YuviPanda>	 hmm
[14:43:03] <YuviPanda>	 andrewbogott: no, there isn't. the second patch adds one, I'm modifying it now
[14:43:12] <YuviPanda>	 Coren: hmm,  symlink for now from /var/lib/carbon to /srv/carbon then?
[14:44:23] <Coren>	 That'd work, even though it's a little cruddy.  It would have the advantage that the difference lives in puppet without having to configure graphite any fifferently.
[14:44:28] <Coren>	 differently*
[14:45:49] <YuviPanda>	 Coren: yeah, let me do that
[14:46:14] <YuviPanda>	 andrewbogott: can you find out the user / group of /var/lib/carbon on tungsten?
[14:46:48] <andrewbogott>	 drwxr-xr-x  3 _graphite _graphite      39 Jul  8 10:51 .
[14:46:55] <YuviPanda>	 hmm _graphite
[14:46:59] <YuviPanda>	 ?!
[14:47:13] <YuviPanda>	 indeed
[14:51:08] <^d>	 gah, who f'd beta?
[14:51:25] <^d>	 Now I can't even run a maintenance script at all.
[14:51:26] <^d>	 http://p.defau.lt/?IOa_VAbuwPVLTD0K7BcuIA
[14:54:06] <YuviPanda>	 ^d: blame bd808 
[14:54:36] <bd808>	 What did I do this time?
[14:54:57] <^d>	 wikiversions + symlinks on beta are wonky.
[14:55:03] <^d>	 and I can't run maintenance scripts.
[14:55:13] <bd808>	 "Undefined variable: db"
[14:56:19] <bd808>	 I'll look. Hacks in beta got messed up when somebody had the audacity to make puppet actually setup a prod host correctly.
[14:56:19] <^d>	 Poor error handling in MWMultiVersion
[14:56:25] <^d>	 But basically because it can't read the CDB file.
[14:58:19] <^d>	 Fixed.
[14:58:25] <^d>	 php was still pointing at wrong place.
[14:58:28] <^d>	 silly symlinks.
[14:58:43] <bd808>	 Did we both fix it at the same time? :)
[14:58:54] <bd808>	 `mv common-local common-local.bak; ln -s /srv/scap-stage-dir common-local`
[14:59:15] <bd808>	 I guess we will see if that survives the next puppet run
[14:59:46] <^d>	 No, I think you did that.
[14:59:50] <^d>	 I did something else I spotted.
[14:59:55] <^d>	 in /a/common on -bastion
[15:03:05] <hashar>	 !log deployment-prep Killed some stalled scap / rsync process on deployment-bastion that were preventing https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ from acquiring the lock.
[15:03:13] <labs-morebots>	 Logged the message, Master
[15:31:15] <MC8>	 hm, getting 502 on https://tools.wmflabs.org/guc
[15:43:36] <JohnLewis>	 !log mailman create mailman-03 using precise
[15:43:38] <labs-morebots>	 Logged the message, Master
[15:44:56] <bd808>	 !log deployment-prep /var full on deployment-mediawiki02; deleting 572M /var/log/apache2/debug.log.1
[15:44:58] <labs-morebots>	 Logged the message, Master
[15:47:57] <bd808>	 !log deployment-prep Changed apache logging level from debug to warn on deployment-mediawiki02
[15:48:00] <labs-morebots>	 Logged the message, Master
[15:52:54] <bd808>	 !log deployment-prep Changed apache logging level from debug to notice on deployment-mediawiki02 in /usr/local/apache/conf/wmflabs-logging.conf
[15:52:57] <labs-morebots>	 Logged the message, Master
[16:03:07] <bd808>	 !log deployment-prep Removed local changes to /usr/local/apache/conf/wmflabs-logging.conf on deployment-mediawiki02; logs back to nfs share
[16:03:09] <labs-morebots>	 Logged the message, Master
[16:11:39] <hashar>	 !log deployment-prep deleted /usr/local/apache/common-local symlink, made it a directory and retriggered https://integration.wikimedia.org/ci/job/beta-scap-eqiad/17887/console
[16:11:41] <labs-morebots>	 Logged the message, Master
[16:16:19] <bd808>	 hashar: Are you moving thing to the "real" directory? I put the symlink back this morning because ^d was not able to run mwscript commands on deployment-bastion
[16:16:38] * bd808 didn't mean to step on toes
[16:18:15] <^d>	 whoo, consistency.
[16:19:03] <bd808>	 Getting rid of the hacks I made for scap in beta will be good. I hope we have enough disk though
[16:29:34] <hashar>	 bd808: ah man sorry :(
[16:29:35] <hashar>	 what a mess
[16:29:52] <hashar>	 I have made it a directory which has been synced https://integration.wikimedia.org/ci/job/beta-scap-eqiad/17887/console
[16:30:03] <hashar>	 cause I had a bunch of permissions denied in a previous run sorry
[16:30:12] <hashar>	 perm errors : https://integration.wikimedia.org/ci/job/beta-scap-eqiad/17886/console
[16:30:16] <bd808>	 hashar: No worries. If you are on top of it I'll keep my hands off
[16:30:28] <hashar>	 na I am attending the weekly checkin then back home
[16:30:47] <hashar>	 and I am not entirely sure where to point things :/  All those directories / symlinks are terribly confusing me
[16:31:16] <bd808>	 The apps expect /usr/local/apache/common-local
[16:31:25] <bd808>	 everything else is just scap handwaving
[16:31:56] <bd808>	 Most hosts have a secondary disk mount at /srv so I put stuff on that
[16:32:33] <bd808>	 on bastion the disk is mounted a /mnt with /srv being a symlink to /mnt/srv so we wouldn't run out of space on the primary dirve
[16:32:44] <bd808>	 But it was a tangle for sure
[16:32:46] <hashar>	 ahh
[16:33:10] <hashar>	 the beta-scap-eqiad job also times out on scap-rebuild-cdbs (toooo many messages! :D )
[16:33:49] <bd808>	 The first sync will take a while for sure
[16:34:32] <hashar>	 oh success
[16:35:17] <hashar>	 code update running now
[16:38:53] <hashar>	 and triggered https://integration.wikimedia.org/ci/job/beta-scap-eqiad/17888/console
[16:44:26] <gifti>	 i could swear there was a bug that the labs dumps are outdated. where is it?
[16:51:58] <gifti>	 duh, found it; it is totally unappropriately named 9_9
[16:57:00] * hashar disappears
[17:58:53] <grrrit-wm1>	 (03CR) 10Merlijn van Deen: [C: 032 V: 032] "Sorry for not merging this sooner!" [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143239 (owner: 10Jforrester)
[18:05:07] <James_F>	 YuviPanda: Ping re. https://gerrit.wikimedia.org/r/#/c/143238/
[18:06:10] <YuviPanda>	 James_F: I think 'solution' there is to add feature to wikibugs that lets it spew things in more than one channel...
[18:06:52] <James_F>	 YuviPanda: OK. Who's going to write that, and are you going to either fix the existing issues or let this merge in the mean time? Right now those bugs are just being ignored because they're going to the wrong channel…
[18:07:28] <YuviPanda>	 James_F: I'll go write that now, and then I'll poke you :)
[18:08:08] <James_F>	 YuviPanda: Sounds great. :-)
[18:08:14] <James_F>	 YuviPanda: Sorry to bug you!
[18:09:23] <valhallasw`cloud>	 YuviPanda: it can do that. I think.
[18:09:28] <YuviPanda>	 valhallasw`cloud: oh?
[18:09:50] <YuviPanda>	 valhallasw`cloud: indeed, it can!
[18:09:53] <valhallasw`cloud>	 just add the project to two channels
[18:09:55] <valhallasw`cloud>	 no problem
[18:10:05] <YuviPanda>	 James_F: ^
[18:10:09] <valhallasw`cloud>	 having it both in default_channel /and/ somewhere else is slightly more work
[18:10:14] <YuviPanda>	 valhallasw`cloud: why so?
[18:10:31] <YuviPanda>	 valhallasw`cloud: you just specify the default channel as another channel as well
[18:10:36] <valhallasw`cloud>	 because we then need to add the default_channel back in the channel config
[18:10:36] <valhallasw`cloud>	 yeah
[18:10:44] <YuviPanda>	 valhallasw`cloud: not *that* much more :)
[18:10:57] <valhallasw`cloud>	 \epsilon
[18:11:04] <YuviPanda>	 James_F: I'll remove the -2 if you can amend it to also include #wikimedia-dev
[18:11:08] <valhallasw`cloud>	 I'm also trying to find out why it's not responding to CTCPs
[18:12:02] <valhallasw`cloud>	 anyway, let me deploy 239
[18:13:07] <valhallasw`cloud>	 now it does work. weirddddd
[18:31:32] * YuviPanda pokes Coren
[18:31:36] <YuviPanda>	 poke back when you have a few minutes?
[18:34:10] <James_F>	 YuviPanda: Aha, OK.
[18:38:57] <grrrit-wm1>	 (03PS4) 10Jforrester: Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 
[18:39:51] <grrrit-wm1>	 (03PS5) 10Jforrester: Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 
[18:40:03] <grrrit-wm1>	 (03CR) 10Jforrester: "PS4 addresses Yuvi's concerns; PS5 is a rebase." [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester)
[18:41:19] <valhallasw`cloud>	 YuviPanda: shouldn't just all mediawiki/* bugs be in #wikimedia-dev? e.g. also "File management", "Uploading"
[18:41:57] <James_F>	 valhallasw`cloud: That was Nemo_bis's suggestion; I invited him to submit a patch if he wanted it to be done.
[18:42:27] <valhallasw`cloud>	 ok
[18:42:49] <grrrit-wm1>	 (03PS6) 10Merlijn van Deen: Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester)
[18:43:15] <James_F>	 Err.
[18:43:31] <James_F>	 https://gerrit.wikimedia.org/r/#/c/143238/5..6/channels.py
[18:43:37] <James_F>	 That presumably wasn't what you intended?
[18:43:50] <James_F>	 Did you alter based on PS4 rather than PS5?
[18:44:05] <valhallasw`cloud>	 ok
[18:44:06] <valhallasw`cloud>	 oh
[18:44:26] <valhallasw`cloud>	 yes :-(
[18:44:30] * James_F fixes.
[18:45:48] <grrrit-wm1>	 (03PS7) 10Jforrester: Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 
[18:46:07] <valhallasw`cloud>	 thanks
[18:46:13] <James_F>	 No worries. :-)
[18:46:49] <grrrit-wm1>	 (03CR) 10Merlijn van Deen: [C: 032 V: 032] Add some extra extensions and bug areas for the Editing team [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/143238 (owner: 10Jforrester)
[18:47:05] <James_F>	 valhallasw`cloud: Won't work; -2s trump +2s.
[18:47:17] <James_F>	 Oh, you deleted YuviPanda's -2. :-)
[18:47:19] <James_F>	 Never mind!
[18:47:23] <valhallasw`cloud>	 [x] next to the reviewer trumps -2 ;-)
[18:47:25] <YuviPanda>	 hehe :)
[18:47:28] * James_F grins.
[18:47:38] <valhallasw`cloud>	 lemme deploy that
[18:47:38] <YuviPanda>	 OH GOD I MUST START DRAMA NOW
[18:47:57] * YuviPanda emails wikimedia-l and wikitech-l a copy of one of Shakespeare's plays
[18:49:18] <mutante>	 waits for spamassassin to be done checking Yuvi's mail :p
[18:51:37] <grrrit-wm1>	 (03PS1) 10Merlijn van Deen: fix syntax [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/155095 
[18:51:58] <grrrit-wm1>	 (03CR) 10Merlijn van Deen: [C: 032 V: 032] fix syntax [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/155095 (owner: 10Merlijn van Deen)
[18:52:11] <James_F>	 Ha. Whoops.
[18:52:47] <valhallasw`cloud>	 that was the most painful quickfix /evah/
[18:53:04] <James_F>	 :-)
[18:53:06] <valhallasw`cloud>	 because pushing to gerrit from within tools is a PITA :-p
[18:53:14] <James_F>	 Lazy.
[18:55:15] <^d>	 Can do it over https.
[18:55:26] <^d>	 Makes it easier than key forwarding and so-forth.
[18:58:46] <Coren>	 YuviPanda: I have a bit 'o' time.  What be up?
[18:59:04] <YuviPanda>	 Coren: networking setup for letting labs instances send data to labmon1001.
[18:59:10] <Nemo_bis>	 ...
[18:59:14] <YuviPanda>	 Coren: similar to what we have for labsdb, but different ports, I suppose.
[18:59:15] <mutante>	 valhallasw`cloud: what ^d says.. should work once you set the "http password" in gerrit ui afaik
[18:59:34] <YuviPanda>	 Coren: I couldn't find it on puppet. I suppose it might be on the networking hardware?
[18:59:50] <Coren>	 YuviPanda: Ah.  I'm very much the wrong person to ask that - I summon help via magic RT tickets.  :-)
[19:00:02] <Coren>	 Yes, it needs router surgery.
[19:00:03] <YuviPanda>	 Coren: aaah, ugh. I shall file one, then. who usually handles those?
[19:00:07] <mutante>	 heh, i said the same, thinking it is on hardware routers
[19:00:12] <mutante>	 re: NAT rules
[19:00:24] <mutante>	 YuviPanda: role::network_engineer
[19:00:28] <YuviPanda>	 hehe
[19:00:48] <mutante>	 networking queue in RT, actually
[19:01:00] <Coren>	 YuviPanda: Depends.  Mark very often though I expect he'll be inordinately busy for a bit.  Faidon also normally does a lot of it.
[19:01:04] <YuviPanda>	 it was labs, so I presumed it would be more in openstack
[19:01:23] <Coren>	 YuviPanda: Many can do certain parts.  Open the RT ticket and it'll get love by whoever is available.
[19:02:40] <YuviPanda>	 Coren: cool. graphite is up and running on labmon1001, btw
[19:03:04] <valhallasw`cloud>	 mutante: does git-review understand that? :-p
[19:03:10] <Coren>	 Yeay graphite!
[19:06:10] <valhallasw`cloud>	 it seems it does :-)
[19:06:52] <valhallasw`cloud>	 !log wikibugs new version deployed (gerrit 143239 and 143238)
[19:06:53] <labs-morebots>	 wikibugs is not a valid project.
[19:06:56] <valhallasw`cloud>	 !log tools.wikibugs new version deployed (gerrit 143239 and 143238)
[19:06:58] <labs-morebots>	 Logged the message, Master
[19:09:21] <grrrit-wm1>	 (03PS1) 10Merlijn van Deen: Add 'emergency fixes' to readme [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/155098 
[19:09:26] <valhallasw`cloud>	 ^ seems to work :-)
[19:09:38] <grrrit-wm1>	 (03CR) 10Merlijn van Deen: [C: 032 V: 032] Add 'emergency fixes' to readme [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/155098 (owner: 10Merlijn van Deen)
[19:09:45] * James_F grins.
[19:10:48] <valhallasw`cloud>	 document ALL the things :-p
[19:12:39] <chippy>	 hello, I'm working on the maps warper application for commons, and want to use OAuth for authentication. I'm only able to see the oauth consumer registration page on wikitech but not on other wikis, what should I do?
[19:13:29] <valhallasw`cloud>	 chippy: use the one on mediawiki.org
[19:13:43] <valhallasw`cloud>	 that one gives access to all WMF wikis that use centralauth
[19:15:29] <chippy>	 thanks valhallasw`cloud looking now
[19:16:09] <chippy>	 hmm, I cannot find the registration page on meta either
[19:16:30] <chippy>	 it could be that my wikitech account and global auth are different?
[19:17:11] <mutante>	 chippy: yes, wikitech is labs LDAP 
[19:17:33] <mutante>	 and global wikimedia (SUL) account is not identical
[19:17:37] <chippy>	 okay that's cool
[19:18:21] <mutante>	 make a new one on wikitech to get labs access 
[19:18:28] <chippy>	 I've got labs access
[19:18:47] <valhallasw`cloud>	 chippy: not on meta, on mediawiki.org :-p
[19:19:04] <valhallasw`cloud>	 https://www.mediawiki.org/wiki/Special:OAuthConsumerRegistration
[19:19:15] <chippy>	 aAHH
[19:19:22] <chippy>	 yes! Thank you valhallasw`cloud 
[19:20:18] <chippy>	 that's great.
[19:29:59] <^d>	 I built an instance but I think I forgot to open the port I needed from the security groups. Ways around rebuilding?
[19:30:37] <^d>	 This is a massively unpuppetized test and I'd rather not spend another half-day getting back where I am now.
[19:32:17] <Coren>	 ^d: No, but you can always open the port in the default security group.
[19:32:39] <^d>	 Long as I remember to remove it later, this is deployment-prep :p
[19:35:07] <^d>	 How long I have to wait?
[19:35:26] <Coren>	 Wait for what?
[19:35:32] <^d>	 The rule to take effect?
[19:35:40] <Coren>	 That's basically instantaneous.
[19:35:57] <^d>	 Oh there it goes.
[19:36:05] <^d>	 I should've waited like 2-3 seconds :p
[20:21:24] <mutante>	 bd808: do you connect to another bastion to get from there to deployment-bastion?
[20:21:27] <mutante>	 2 bastion hop?
[20:24:15] <bd808>	 mutante: The labs bastion. deployment-bastion isn't really a bastion. It's more a clone of tin.
[20:24:24] <bd808>	 With a bad name
[20:25:00] <mutante>	 bd808: i see, ok:)
[20:25:10] <mutante>	 so labs bastion is also different for ops
[20:25:17] <mutante>	 but that command works
[20:26:48] <bd808>	 thanks for the merge.
[20:27:55] <mutante>	 np, wait for submit...
[20:28:41] <mutante>	 that was a good point re: LDAP auth for service inside labs
[20:28:46] <mutante>	 but too bad
[20:29:08] <mutante>	 i'd have another project where i could use Apache auth and not have separate passwords again
[20:33:18] <mutante>	 bd808: and now it's actually merged
[20:33:46] <bd808>	 mutante: Yeah. We need a solution for this issue.
[20:33:58] <bd808>	 Ryan and I talked about it many months ago.
[20:34:16] <bd808>	 He had an idea to add a openid provider to wikitech that we could use
[20:34:22] <bd808>	 but that has never been done
[20:34:47] <bd808>	 I think there is a bug for it somewhere...
[20:35:46] <bd808>	 mutante: https://bugzilla.wikimedia.org/show_bug.cgi?id=61754
[20:36:50] <mutante>	 bd808: ah, existing bug, nice :)
[20:40:06] <mutante>	 bd808: i think the scrubbing it is
[20:40:42] <mutante>	 and then kill the need for pass entirely
[20:41:05] <bd808>	 That would be nice. scrubbing logs is tricky business however
[20:41:26] <bd808>	 ips, passwords and god knows what else can be in messages and stack traces
[20:42:44] <mutante>	 yea, but it sounds like the problem with db replication
[20:45:16] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Low-quality images are not rendered in beta - 10https://bugzilla.wikimedia.org/69757 (10Yuri Astrakhan) 3NEW p:3Unprio s:3major a:3None This shows: http://upload.beta.wmflabs.org/wikipedia/en/thumb/4/4d/Snowman.JPG/120px-Snowman.JPG  This doesn't: http://upl...
[20:52:42] <scfc_de>	 bd808, mutante: Can't you use OAuth that YuviPanda uses for quarry.wmflabs.org?
[20:53:07] <akoopal>	 guc is not responding, it's webserver overloaded?
[20:53:33] <bd808>	 scfc_de: Tricky. I really need the auth to be handled by apache for that app.
[20:54:18] <bd808>	 I don't know if there is a mod_auth[nz] for oauth
[20:59:00] <mutante>	 N: Ignoring file 'puppet_base_2.7' in directory '/etc/apt/preferences.d/' as it has an invalid filename extension
[20:59:19] <bd808>	 That was cruft from the puppet 3 migration
[20:59:38] <mutante>	 delete manually? hmm..ok
[21:00:07] <mutante>	 i guess i should make a new instance entirely before i touch the mariadb install issue
[21:00:25] <mutante>	 also  apache2 : Depends: apache2-mpm-worker (= 2.2.22-1ubuntu1.7) but it is not going to be installed or
[21:00:51] <scfc_de>	 bd808: Probably not :-).
[21:01:06] <scfc_de>	 *mod_auth[nz] for oauth existing
[21:01:18] <bd808>	 mutante: hmmm... I remember that vaguely. I thought it was fixed though
[21:01:42] <bd808>	 It was something about the apache class changes a month or so ago
[21:01:43] <mutante>	 i guess it depends what you mean by fixed
[21:01:52] <mutante>	 if that is "make a new instance and move your stuff" it might
[21:02:00] <mutante>	 but existing project is broken
[21:02:12] <bd808>	 VMs are cattle , not pets
[21:02:19] <mutante>	 yea, and i get those issues now because puppet did not run for so long :p
[21:03:11] <mutante>	 yea, but do i have to recreate it every couple weeks for this reason?
[21:03:16] <bd808>	 Somebody should make better labs monitoring the #1 priority for YuviPanda :)
[21:04:01] <YuviPanda>	 bd808: graphite is running on labmon1001 fully now :) Need to setup NAT rules, and then we can have graphite. 
[21:04:26] <YuviPanda>	 bd808: it'll be '#1 priority' when I 'fully' move to Ops, I suppose
[21:04:33] <YuviPanda>	 which isn't for another few months
[21:04:42] <mutante>	 DPKG check would be nice too
[21:04:42] <bd808>	 slacker
[21:04:44] <bd808>	 :)
[21:05:12] <mutante>	 wonders if he should give up on Icinga
[21:05:18] <mutante>	 i cant test it on puppet compiler 
[21:05:23] <mutante>	 because of naggen
[21:05:50] <mutante>	 then if we replace it it would be shinken...
[21:05:56] <mutante>	 again unrelated to labs :)
[21:07:51] <alkamid>	 Has something happened to /public/dumps/public? The most recent plwiktionary dump I can see there is from 20140624, but there is one from 20140813 at http://dumps.wikimedia.org/plwiktionary/
[21:08:22] <mutante>	 apergos will know
[21:10:33] <scfc_de>	 alkamid: https://bugzilla.wikimedia.org/show_bug.cgi?id=66362
[21:14:27] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Low-quality images are not rendered in beta - 10https://bugzilla.wikimedia.org/69757#c1 (10Antoine "hashar" Musso) I think that is handled by deployment-upload.eqiad.wmflabs which is more or less a simulation of the old media servers using nginx.  IIRC thumb.php got...
[21:14:42] <mutante>	 !log wikistats - puppet broken, Apache setup, DB setup, DPKG, attempting to revive on fresh instance wikistats-petcow
[21:14:45] <labs-morebots>	 Logged the message, Master
[21:15:39] <alkamid>	 scfc_de, thanks, but they don't really say why the dumps are dead or if something is being done about it
[21:16:12] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Make the password for logstash-beta.wmflabs.org available to all users with sudo - 10https://bugzilla.wikimedia.org/69267 (10Bryan Davis) 5PATC>3RESO/FIX
[21:17:01] <mutante>	 alkamid: i think the "why" is "not enough disk space"
[21:18:14] <mutante>	 eh, at least the bug title makes it sound that way
[21:21:10] <mutante>	 what are the puppet groups for mariadb and apache nowadays?
[21:22:44] <mutante>	 Error: Could not set 'directory' on ensure: ... hrmm
[21:23:15] <mutante>	 Warning: /Stage[main]/Apache/Service[apache2]: Skipping because of failed dependencies
[21:23:41] <mutante>	 do we have any working puppet group that installs a webserver?
[21:25:02] <mutante>	 how do you create systemusers with puppet now?
[21:40:58] <mutante>	 ok, one by one. how to you create system users on labs instances?
[21:42:01] <mutante>	 user{} and group{} don't appear to do it anymore
[21:42:14] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): easily reload all apaches - 10https://bugzilla.wikimedia.org/36422#c19 (10Bryan Davis) 5PATC>3NEW a:5Bryan Davis>3None My patch was merged, but puppet<->salt integration is disabled/broken due to race conditions that occur when new instances are added and ne...
[21:44:35] <Thehelpfulone>	 !log mailman recreated mailman-03 m1.small with mailman, apache2 and lighttpd and precise
[21:44:36] <labs-morebots>	 Logged the message, Master
[21:45:41] <mutante>	 (/Stage[main]/Wikistats/Group[wikistatsuser]/ensure) created
[21:45:43] <mutante>	 Could not create user wikistatsuser: Execution of '/usr/sbin/useradd -g wikistatsuser -G wikistats -d /usr/lib/wikistats -m -r wikistatsuser' returned 6: useradd: group 'wikistats' does not exist
[21:46:41] <mutante>	 ah, wrong group name, found on e:)
[22:26:17] <mutante>	 are there any plans about having an apt.wm.org for labs only?
[22:26:40] <mutante>	 i would like to import a deb and install it from an instance, but it probably doesnt belong in prod
[22:27:41] <YuviPanda>	 mutante: labsdebrepo :)
[22:27:55] <mutante>	 YuviPanda: oh oh,, how to import 
[22:27:57] <mutante>	 looks
[22:28:22] <mutante>	 reads Help page, thanks
[22:28:27] <YuviPanda>	 mutante: \o/
[23:15:03] <YuviPanda>	 valhallasw`cloud: is there a known set of toollabs IPs that edit wiki?
[23:15:06] <YuviPanda>	 legoktm: ^
[23:15:15] <legoktm>	 I don't think so
[23:15:43] <legoktm>	 I think it could be anywhere in 10.0.0.0/8
[23:16:33] <YuviPanda>	 legoktm: oh, that's fine too
[23:17:07] <legoktm>	 there have been a few instances where some config was wrong and random edits from outside came from internal IPs too
[23:17:19] <YuviPanda>	 heh
[23:19:04] <Merlissimo>	 YuviPanda: 2620:0:860:0:0:0:0:0/46 for ip6
[23:24:04] <YuviPanda>	 Merlissimo: aaah, cool, thanks :)
[23:24:51] <Merlissimo>	 source https://bugzilla.wikimedia.org/show_bug.cgi?id=56681
[23:25:19] <Merlissimo>	 this range was added to autoblick whitelist on multiple wmf wikis
[23:25:40] <YuviPanda>	 Merlissimo: aah, that was useful. thanks!