[00:57:19] <superm401>	 I'm getting sporadic 503 errors on bits:
[00:57:31] <superm401>	 E.g. http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php?debug=false&lang=en&modules=ext.centralNotice.bannerController|ext.centralauth.centralautologin|ext.uls.init%2Cinterface%2Cpreferences%2Cwebfonts|ext.visualEditor.viewPageTarget.init|jquery.accessKeyLabel%2CbyteLength%2Cclient%2Ccookie%2CmwExtension%2CtabIndex%2Cthrottle-debounce%2Ctipsy|mediawiki.Title%2CUri%2Capi%2Ccldr%2CjqueryMsg%2Clanguage%2Cnotify%2Cuser%
[00:57:32] <superm401>	 2Cutil|mediawiki.language.data%2Cinit|mediawiki.legacy.ajax%2Cwikibits|mediawiki.libs.pluralruleparser|mediawiki.page.startup|mmv.base%2Chead|mw.TMHGalleryHook.js|skins.vector.js&skin=vector&version=20140909T200920Z&*
[00:57:52] <superm401>	 http://goo.gl/BMxWqH
[00:57:58] <superm401>	 That redirects to a 503 for me.
[02:05:14] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): [OPS] debianize PHP5 extension 'parsekit' - 10https://bugzilla.wikimedia.org/37076#c15 (10Daniel Zahn) on integration slaves in labs it is attempted to install this package but it fails:  Error: /Stage[main]/Contint::Packages/Package[php5-parsekit]/ensure: change fr...
[02:11:19] <mutante>	 !log integration - package php5-parsekit not found on trusty slave
[02:11:25] <labs-morebots>	 Logged the message, Master
[02:14:29] <mutante>	 !log integration package nodejs-legacy not found - puppet fail on integration slaves
[02:14:34] <labs-morebots>	 Logged the message, Master
[08:03:42] <Krinkle>	 superm401: The error message is quite generic, could be a routing problem. It's not from MediaWiki itself I think
[08:04:12] <Krinkle>	 It applies to all of labs bits though, even simple urls: http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php?debug=false&lang=en&modules=foo 
[08:04:13] <jeremyb>	 you mean cronspam?
[08:04:19] <Krinkle>	 http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php
[08:04:34] <jeremyb>	 oh
[08:04:45] <Krinkle>	 > superm401: [2014-09-10 00:55:05 +00:00] I'm getting sporadic 503 errors on bits: 
[08:06:29] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): [OPS] debianize PHP5 extension 'parsekit' - 10https://bugzilla.wikimedia.org/37076#c16 (10Krinkle) See bug 68256 comment 5. The php jobs currently don't run on the trusty slaves so it missing will not cause immediate problems, and puppet fortunately continues applyi...
[08:07:17] * jeremyb is looking
[08:12:03] * jeremyb waits for puppet
[08:12:35] <jeremyb>	 errr, i mean gerrit
[08:12:38] <jeremyb>	 puppet repo
[08:14:07] <Krinkle>	 !log deployment-prep bits.beta.wmflabs.org is down with 503 Service Unavailable (http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php)
[08:14:09] <labs-morebots>	 Logged the message, Master
[08:14:17] <Krinkle>	 k, have fun fixing that. I'm boarding my flight.
[08:14:34] <Krinkle>	 seems to have been down for about 10 hours
[08:15:24] <Krinkle>	 greg-g: superm401: YuviPanda|bzzzz: Reedy: ^
[08:30:56] <jeremyb>	 superm401: it's not just bits. en is down too
[08:32:58] <jeremyb>	 and the docroot does in fact not exist on disk
[08:33:07] <jeremyb>	 i don't know why you'd expect that to work
[08:33:10] <jeremyb>	 :)
[08:38:59] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: [Regression] WMFLabs: Unable to delete any instance - 10https://bugzilla.wikimedia.org/70636 (10Andre Klapper) p:5Unprio>3High s:5normal>3major
[08:39:14] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: [Regression] WMFLabs: Nova project quota broken - 10https://bugzilla.wikimedia.org/70634 (10Andre Klapper) p:5Unprio>3High
[10:31:51] <dan-nl>	 anyone happen to know why commons beta is down? http://commons.wikimedia.beta.wmflabs.org
[10:32:51] <jeremyb>	 dan-nl: #wikimedia-qa
[11:20:44] <GerardM->	 http://tools.wmflabs.org/reasonator/?q=1402277
[11:21:01] <GerardM->	 it looks as if it cannot read the database
[11:37:53] <GerardM->	 magnus fixed it
[12:49:59] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: Wikitech: Performing content actions results in PHP strict warning by MWSearch outputted on the page - 10https://bugzilla.wikimedia.org/70436#c3 (10Filippo Giunchedi) got the same error while moving a page https://wikitech.wikimedia.org/w/index.php?title=Incident_docume...
[13:23:08] <a930913>	 Coren: Poke?
[13:26:17] <a930913>	 Need some help working out why inserting a data dump into tools-db is taking so long.
[13:37:30] <andrewbogott>	 woo 333 new emails
[13:38:33] <a930913>	 !email andrewbogott Spam
[13:38:38] * YuviPanda pats andrewbogott
[13:38:49] <a930913>	 More spambots needed here :p
[13:38:57] <YuviPanda>	 I went on vacation for 4 days, only 3 actionable emails
[13:42:28] <a930913>	 I was in a field for 3 weeks, I tried going through my emails as often as possible, but still came back to about 800 unread.
[13:43:14] <YuviPanda>	 hmm
[13:44:13] <andrewbogott>	 YuviPanda: more than 300 of those are from diamond.  "parse error in /etc/sudoers.d/nagios near line 6"  Is that you?
[13:44:30] <YuviPanda>	 andrewbogott: ah, no. maybe godog? i saw some backscroll about that
[13:47:14] <wikibugs>	 3Wikimedia Labs / 3tools: Moving toolserver domain, mail and redirects - 10https://bugzilla.wikimedia.org/66113#c1 (10nosy) Added OSM redirects today. I needed rewrites in our central nginx config that wont come with the regular redirects we place in the user homes. So you'll have to implement those, too.
[13:47:15] <godog>	 YuviPanda: yup it was me this morning :(
[13:47:51] <godog>	 andrewbogott: yep bad sudo puppet change I merged
[13:47:56] <andrewbogott>	 ok, so I can delete and ignore all thos?
[13:48:05] <andrewbogott>	 cool, thanks
[13:48:13] <godog>	 safely so, yes
[13:49:44] <wikibugs>	 3Wikimedia Labs / 3tools: Provide namespace IDs and names in the databases similar to toolserver.namespace - 10https://bugzilla.wikimedia.org/48625#c46 (10nosy) I'd need a DB called toolserver. :D It'd be fine if I can only get toolserver_p. The rest of the DBs of the user can be left untouched.
[13:54:59] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: wikitech strict warnings on API save - 10https://bugzilla.wikimedia.org/70628#c2 (10Andrew Bogott) There is a fix in place now that should resolve those warnings, cherry-picked to the wikitech branch.  Can you please verify if this is correct and, if so, de-hack the bot...
[13:55:00] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: Wikitech: Performing content actions results in PHP strict warning by MWSearch outputted on the page - 10https://bugzilla.wikimedia.org/70436#c4 (10Andrew Bogott) This would be fixed by https://gerrit.wikimedia.org/r/#/c/158390/, but let's fix the actual warning first!
[13:59:58] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: Wikitech: Performing content actions results in PHP strict warning by MWSearch outputted on the page - 10https://bugzilla.wikimedia.org/70436#c5 (10Andrew Bogott) Chad says this is already fixed on master.  So, I'll try to get 158390 rolled out shortly.
[14:24:22] <Coren>	 andrewbogott: Ping?
[14:24:44] <andrewbogott>	 'morning!
[14:25:20] <Coren>	 Heyo.  Switching channels.
[14:39:14] <wikibugs>	 3Wikimedia Labs / 3wikitech-interface: [Regression] WMFLabs: Nova project quota broken - 10https://bugzilla.wikimedia.org/70634#c1 (10Andrew Bogott) I'm seeing some subtle (and not-so-subtle) issues related to dueling auth -- wikitech users have wiki auth and also openstack auth, and sometimes we get into a...
[15:16:30] <andrewbogott>	 wikitech is down again :(  I'm working on it, should be back shortly
[15:19:19] <dan-nl>	 hey all, thatnks for getting commons beta back online
[15:20:22] <dan-nl>	 i'm trying to import the latest Template:Map from commons via an xml export/import process, but unfortunately it errors out with "Import failed: Expected <mediawiki> tag, got". anyone happen to know how to fix that issue?
[15:21:25] <dan-nl>	 also, commons doesn't appear in the transwiki source wiki dropdown list. is there an easy way to add it?
[15:23:34] <grrrit-wm>	 (03PS1) 10Yuvipanda: Kill alpha build folders from previous runs [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/159481 
[15:23:36] <grrrit-wm>	 (03PS1) 10Yuvipanda: Request 8G of RAM for jsub [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/159482 
[15:24:29] <YuviPanda>	 dan-nl: jackmcbarn also had the same bug earlier
[15:24:36] <YuviPanda>	 unsure if he fixed it
[15:24:45] <jackmcbarn>	 nope
[15:24:59] <dan-nl>	 hmm ... k, filing a bug for it
[15:29:16] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Import failed Expected <mediawiki> tag, got - 10https://bugzilla.wikimedia.org/70658 (10dan) 3NEW p:3Unprio s:3normal a:3None commons beta is having a problem importing some templates from commons; e.g. Template:Artwork, Template:Map.   steps to reproduce --...
[15:33:56] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Kill alpha build folders from previous runs [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/159481 (owner: 10Yuvipanda)
[15:34:03] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Request 8G of RAM for jsub [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/159482 (owner: 10Yuvipanda)
[15:34:11] <grrrit-wm>	 (03CR) 10Yuvipanda: [V: 032] Kill alpha build folders from previous runs [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/159481 (owner: 10Yuvipanda)
[15:34:20] <grrrit-wm>	 (03CR) 10Yuvipanda: [V: 032] Request 8G of RAM for jsub [labs/tools/wikipedia-android-builds] - 10https://gerrit.wikimedia.org/r/159482 (owner: 10Yuvipanda)
[15:47:14] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Import failed Expected <mediawiki> tag, got - 10https://bugzilla.wikimedia.org/70658#c1 (10Jackmcbarn) This also happens when doing interwiki imports from en.wikipedia.org to test.wikipedia.org. Note that the export files themselves are fine, as I was able to import...
[15:52:44] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster api.php, index.php, load.php return 404 (caused failed browser tests) - 10https://bugzilla.wikimedia.org/70648 (10Greg Grossmeier) p:5Unprio>3Highes
[15:54:56] <tonythomas>	 where will I get the pupet config repo inside a labs instance which is having  role::puppet::self [] enabled ?
[15:55:16] <YuviPanda>	 tonythomas: /var/lib/git/operations/puppet
[15:55:50] <tonythomas>	 oops :\ cd: /var/lib/git/operations: No such file or directory
[15:56:17] <tonythomas>	 even though  role::puppet::self [?] is ticked in my host 'verpwebhost' inside mediawiki-verp project 
[15:56:23] <tonythomas>	 the puppet status is stale too 
[15:56:52] <YuviPanda>	 tonythomas: you need to run puppet after ticking that as well
[15:56:55] <YuviPanda>	 sudo puppet agent -tv
[15:57:11] <tonythomas>	 now its stale, running it will work ?
[16:00:16] <YuviPanda>	 tonythomas: yeah, the wikitech puppet status is kinda useless
[16:00:18] <YuviPanda>	 and out of date
[16:00:20] <YuviPanda>	 and inaccurate
[16:00:23] <YuviPanda>	 and all the bad things
[16:02:22] <tonythomas>	 k :) let me try that again
[16:03:13] <a930913>	 scfc_de: I recall from a Wikimania prehack, that the main reason that histroapedia wasn't on the labs, was because it ran on windoze.
[16:05:28] <scfc_de>	 a930913: Oh, I thought they were trying to monetize it.
[16:06:10] <tonythomas>	 YuviPanda: on a new instance, I get this Error: /Stage[main]/Role::Labs::Instance/Mount[/data/project]: Could not evaluate: Execution of '/bin/mount /data/project' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/mediawiki-verp/project failed, reason given by server: No such file or directory on running puppet agent -tv
[16:06:22] <YuviPanda>	 yeaaah, that's fine, I think
[16:06:25] <YuviPanda>	 ignore that as well
[16:06:32] <tonythomas>	 oeky. in that case. :) 
[16:07:24] <a930913>	 scfc_de: They were I think, but only with regards to providing it as a service to governments and suchlike, for educational purposes.
[16:11:35] <tonythomas>	 YuviPanda: so -> if I have a patch pointing to operations/puppet in gerrit (https://gerrit.wikimedia.org/r/#/c/155753/ ) and I want to test the puppet output to exim in that labs instance, I can apply the patch in /var/lib/git/operations/puppet and do puppet-apply again ?
[16:12:03] <YuviPanda>	 tonythomas: yeah
[16:14:32] <scfc_de>	 tonythomas: The Gerrit page should also have a command line that you can just copy & paste after cd'ing into /var/lib/git/operations/puppet ("git fetch $SOMETHING && git checkout FETCH_HEAD" or something like that).
[16:14:50] <tonythomas>	 scfc_de: true did the anonymous fetch 
[16:15:02] <tonythomas>	 HEAD is now at f891b69... Added the bouncehandler router to catch in all bounce email
[16:15:08] <tonythomas>	 now the puppet-apply right ?
[16:16:34] <Betacommand>	 Coren: any updates on the spidering?
[16:17:06] <YuviPanda>	 tonythomas: yeah, puppet agent -tv again
[16:17:30] <tonythomas>	 YuviPanda: ok. trying that 
[16:19:58] <tonythomas>	 YuviPanda: I did, but got a lot of connection errors listed, and the change is not visible in /etc/exim4/exim4.conf.template
[16:20:22] <YuviPanda>	 tonythomas: ah, try 'sudo service puppet start' and then apply?
[16:22:28] <tonythomas>	 YuviPanda: did that. even restarted exim4, but the change is not yet there ! 
[16:22:41] <tonythomas>	 should I add some exim role from wikitech ?
[16:22:49] <YuviPanda>	 tonythomas: oh, yeah, right
[16:23:05] <YuviPanda>	 tonythomas: you could either do that, or 'include <rolename>' under 'node default {' in manifests/site.pp
[16:23:18] <YuviPanda>	 tonythomas: but after the puppet service start you shouldn't be getting connection errors
[16:25:06] <tonythomas>	 YuviPanda: but it looks like I still get connection errors after sudo service puppet start
[16:25:20] <YuviPanda>	 uh
[16:25:23] <YuviPanda>	 that's weird :|
[16:25:33] <YuviPanda>	 try doing a sudo service puppet restart instead of start?
[16:26:24] <tonythomas>	 YuviPanda: did that https://dpaste.de/kF9X
[16:26:35] <tonythomas>	 and puppet-status in wikitech page says - failed 
[16:26:38] <YuviPanda>	 hmm
[16:26:43] <YuviPanda>	 which project is this?
[16:26:49] <YuviPanda>	 can you add me to it and tell me the instance name?
[16:27:02] <tonythomas>	 mediawiki-verp 
[16:27:08] <tonythomas>	 will add ya in a sec
[16:28:47] <tonythomas>	 YuviPanda: your username is Yuvipanda ?
[16:28:52] <YuviPanda>	 ya
[16:29:15] <tonythomas>	 I added that. you got some notifs ?
[16:30:00] <YuviPanda>	 tonythomas: what's the name of the instance?
[16:30:13] <tonythomas>	 mediawikiverp
[16:35:31] <YuviPanda>	 tonythomas: hmm, I'm unsure what is happening :(
[16:36:09] <YuviPanda>	 tonythomas: I'm creating another instance
[16:36:30] <tonythomas>	 YuviPanda: okey. looks like the best method 
[16:39:25] <YuviPanda>	 !log mediawiki-verp created instance verpverpverp
[16:39:29] <labs-morebots>	 Logged the message, Master
[16:40:02] <tonythomas>	 YuviPanda: yay. inside it 
[16:40:13] <YuviPanda>	 tonythomas: inside the new instance/
[16:40:15] <YuviPanda>	 ?
[16:40:34] <tonythomas>	 not yet. I meant the config page. I think the login still takes time 
[16:40:40] <YuviPanda>	 ah
[16:40:41] <YuviPanda>	 ok
[16:40:42] <YuviPanda>	 yeah
[16:40:52] <YuviPanda>	 tonythomas: yeah, I created a new instance. Also added myself as projectadmin :)
[16:41:11] <tonythomas>	 k. great. so we enable the exim::ganglia and the puppetmaster::role
[16:41:13] <tonythomas>	 ?
[16:41:21] <tonythomas>	 or just the self puppetmaster?
[16:43:04] <tonythomas>	 ok. now inside verpverpverp
[16:44:14] <YuviPanda>	 tonythomas: aah, you need the role::puppet::self role, not puppetmaster
[16:44:20] <YuviPanda>	 that was the problem with the previous instance, I suppose
[16:45:03] <tonythomas>	 I hope so. ( but I remember ticking self role too ) 
[16:45:15] <YuviPanda>	 tonythomas: also I think exim::ganglia just setsup ganglia monitoring for exim, don't know if it sets up exim itself
[16:45:40] <tonythomas>	 in that case, I think only the self role is needed. right ?
[16:46:05] <YuviPanda>	 yes
[16:46:12] <YuviPanda>	 I already ticked it
[16:47:37] <tonythomas>	 puppet apply under progress right ?
[16:50:29] <YuviPanda>	 tonythomas: ah, it's 'sudo service puppetmaster start' not puppet start
[16:50:35] <YuviPanda>	 tonythomas: and yeah, apply under progress
[16:50:47] <tonythomas>	 now its done, I think. let me give sudo service puppetmaster start
[16:51:01] <YuviPanda>	 tonythomas: already did :) now puppet running from the local repo
[16:51:18] <YuviPanda>	 now you can go ahead and do your testing :)
[16:51:23] <tonythomas>	 yay. now running cool. let me pull in the patch 
[16:57:18] <tonythomas>	 YuviPanda: I saw the changes coming in while the sudo puppet agent -tv was given alright, but still the changes are missing in the exim4.conf though ! :o
[16:57:34] <YuviPanda>	 tonythomas: checking
[16:57:42] <tonythomas>	 templates/exim/exim4.conf.SMTP_IMAP_MM.erb corresponds to somefile inside /etc/exim4/
[16:57:45] <tonythomas>	 right ?
[16:59:15] <YuviPanda>	 tonythomas: so I see that your changes are to role::mail::mx
[16:59:27] <tonythomas>	 YuviPanda: true. 
[16:59:33] <tonythomas>	 so we should get that role too in ?
[16:59:35] <YuviPanda>	 tonythomas: so you should add a 'include role::mail::mx' under node default { at the bottom of manifests/site.pp
[16:59:42] <YuviPanda>	 so it'll be applied
[16:59:46] <YuviPanda>	 otherwise nothing will be applied
[16:59:49] <tonythomas>	 ok. adding that 
[17:03:16] <tonythomas>	 added that, now giving sudo puppet agent -tv
[17:03:28] <tonythomas>	 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Class[Exim4] is already declared in file /etc/puppet/manifests/mail.pp:74; cannot redeclare at /etc/puppet/manifests/role/mail.pp:5 on node verpverpverp.eqiad.wmflabs
[17:03:50] <tonythomas>	 looks like its already there ?
[17:07:08] <tonythomas>	 YuviPanda: looks like we are adding a duplicate. 
[17:07:24] <YuviPanda>	 tonythomas: there's both a manifest and a role, I'm not really sure how this is organized
[17:07:38] <YuviPanda>	 I don't know where the manifest include is coming from either 
[17:07:45] <YuviPanda>	 and I've to go now as well :(
[17:07:48] <YuviPanda>	 sorry!
[17:08:13] <tonythomas>	 YuviPanda: ok. thanks for your time anyway, we are a lot far from the start :) 
[17:08:21] <YuviPanda>	 \o/
[17:12:20] <scfc_de>	 Does someone know why spammers like https://wikitech.wikimedia.org/wiki/User:Alex_brollo/OpalToIA that much?! :-)
[17:21:57] <GerardM->	 FYI I understand that statistics are complete ... I noticed that Magnus's stats got updated without him lifting a finger :)
[17:22:23] <GerardM->	 is it all done (and if so) may I ask someone a few questions for a blogpost ?
[17:22:43] <jeremyb>	 andrewbogott: hey. so i'm sure you know beta's out of quota. for instances. but can you bump it up for just today? 1 more instance.
[17:22:45] <GerardM->	 by the way ... this makes for a happy Gerard
[17:23:01] <jeremyb>	 and then i'll give it back
[17:23:42] <jeremyb>	 something fishy and i want a puppet run from clean state
[17:24:31] <andrewbogott>	 jeremyb: we're having some issues with new instances atm, so best if you hold off
[17:24:46] <jeremyb>	 erm?
[17:26:18] <jeremyb>	 oh, you seem to have been doing prod in the other window
[17:28:09] <andrewbogott>	 jeremyb: nfs doesn't like new instances.  We don't know where the problem is and if we turn out to need to retrofit instances that were built during the breakage I want to keep the list short
[17:34:00] <jeremyb>	 andrewbogott: if i don't care about nfs?
[17:34:08] <jeremyb>	 (i'll take local instance $HOME)
[17:35:01] <andrewbogott>	 um… you should care in general since that's where your .ssh keys come from :)
[17:35:07] <andrewbogott>	 Although that part seems not to be broken
[17:35:28] <jeremyb>	 that part also is something that would effect existing instances if it were broken
[17:58:44] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Setup a mediawiki03 (or what not) on Beta Cluster that we can direct the security scanning work to - 10https://bugzilla.wikimedia.org/70181#c9 (10Greg Grossmeier) 13:57 <     bd808> mediawiki03 isn't in the scap pool yet I just noticed. 13:58 <     bd808> so it has...
[17:59:09] <andrewbogott>	 jeremyb: go ahead and create your instance now.
[17:59:19] <jeremyb>	 greg-g: ^
[17:59:19] <andrewbogott>	 Are all 40 of those instances really doing stuff?  Or are half of them left over cruft?
[17:59:38] <jeremyb>	 i have no idea. i just got projectadmin back again yesterday :)
[18:00:46] <greg-g>	 14:00 <     bd808> I think we went on the box killing hunt when siao was added and decided that we  needed all of them
[18:00:57] * greg-g is the copy/paste-r
[18:01:06] <bd808>	 beta keeps getting biggererest
[18:01:46] <jeremyb>	 andrewbogott_afk: anything special to do to not use nfs?
[18:02:04] <jeremyb>	 i guess uncheck in configure
[18:02:11] <bd808>	 It's a project wide flag I think
[18:02:15] <jeremyb>	 although that's a project wide... right
[18:02:29] <jeremyb>	 ok, well either it works or it doesn't (with nfs)
[18:07:18] <Coren>	 jeremyb: Unchecking the box won't do anything if the project directory has already been created.
[18:07:45] <Coren>	 jeremyb: ... but there's very little point to actually somehow disabling NFS.  Just don't write there.  :-)
[18:09:21] <Coren>	 jeremyb: Oh, I read the scrollback.  Don't wory about it; the problem wasn't specifically about NFS and a new instance is probably going to be okay.
[18:09:24] <jeremyb>	 ok
[18:09:37] <jeremyb>	 what was the problem?
[18:11:11] <Coren>	 jeremyb: Something about deleted instances leaving crud in LDAP which, if you are unlucky, may interfere with the config of a new instance.
[18:11:40] <Coren>	 A symptom of which was the NFS server being unable to add the new instance to the ACLs.
[18:11:58] <jeremyb>	 > [1;35merr: /Stage[main]/Role::Labs::Instance/Mount[/home]: Could not evaluate: Execution of '/bin/mount -o rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc /home' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/deployment-prep/home failed, reason given by server: No such file or directory
[18:12:10] <jeremyb>	 on the brand new box
[18:12:15] <Coren>	 When Andrew gave you the heads' up, it was because we didn't yet know what was going on.
[18:12:16] <jeremyb>	 but i'm in
[18:12:19] <jeremyb>	 so it's all ok :)
[18:12:43] <Coren>	 jeremyb: That's the symptom; it may cause other issues - your IP is not "properly" assigned to the instance.  What is the instance name, I can hand-fix it.
[18:12:59] <Coren>	 Wnat what IP does it think it has?
[18:13:45] <jeremyb>	 it can ping the 2 other boxes in that project that it needs to
[18:14:07] <Coren>	 Trust me, you really want me to fix it.  Otherwise you'll get odd problems.
[18:14:23] <jeremyb>	 10.68.17.236
[18:14:25] <jeremyb>	 heh
[18:14:28] <Coren>	 And the instance name?
[18:14:39] <jeremyb>	 deployment-mediawiki04
[18:14:42] <jeremyb>	 or the i name?
[18:15:08] <Coren>	 No, that works.
[18:17:44] <Coren>	 jeremyb: You're all set, but it may take some time before /home and /data/project works (NFS does negative caching for a while)
[18:17:49] <andrewbogott>	 I'm draining the jobqueue on wikitech… my apologies to those receiving a flood of notifications.
[18:18:19] <Coren>	 andrewbogott: Don't we have a cron job that does that?
[18:18:31] <andrewbogott>	 Yep!
[18:18:38] <andrewbogott>	 Seems that it isn't.  I'm investigating.
[19:00:17] <a930913>	 Coren: Poke?
[19:00:24] <Coren>	 Hmm?
[19:00:32] * Coren pokes back! Poing!
[19:00:38] <a930913>	 Coren: Did you see my message above?
[19:01:35] <a930913>	 14:26 < a930913> Need some help working out why inserting a data dump into tools-db is taking so long.
[19:01:48] <Coren>	 Ah, no.  Sorry.
[19:02:42] <Coren>	 But I'm going to venture a guess that you want someone more skilled than I at DBA.  I can see the obvious, but the subtleties of DB efficiencies of mysql require more arcana than I can summon.  :-)
[19:03:11] <a930913>	 Coren: I've bashed a python that extracts metadata from a dump, and puts it into the db.
[19:03:40] <a930913>	 When I htop it, it doesn't look like it's using much resources.
[19:03:40] <Coren>	 Sounds straightforward enough.
[19:04:03] <a930913>	 So I am assuming that the bottleneck is the db.
[19:04:41] <Coren>	 It probably is.  The obvious first question(s) is how you're doing the inserts and whether there are indices, triggers or expensive constraints that might be the primary issue.
[19:04:47] <a930913>	 So is it poor design on my part, or is there too much load on the db?
[19:05:30] <a930913>	 There is one primary key per table, and a few relations.
[19:06:05] <Coren>	 a930913: atm, the DB is fairly busy.  Are you currently inserting things?
[19:06:37] <Coren>	 Funny thing is that it's not I/O bound that I can see, mostly CPU-bound.  Which is odd.
[19:07:05] <a930913>	 (cat /data/project/oar/schema.sql)
[19:07:27] <a930913>	 Coren: Yeah, I've been inserting for about a week. :/
[19:07:45] <a930913>	 (And my computer crashed earlier, so I've lost the mosh session :/ )
[19:08:09] <Coren>	 a930913: What DB are you inserting into?
[19:08:25] <a930913>	 Erm, tools-db wasn't it?
[19:08:48] <Coren>	 Erm, schema.  Sorry, postgres parlance.  :-)
[19:09:12] <Coren>	 I.e.:  s?????_???
[19:09:47] <a930913>	 Coren: s52074__core_metadata
[19:10:55] <Coren>	 a930913: Yeah, I see it.  Hmm.
[19:18:39] <Coren>	 a930913: With postgres, I'd have suggested you use the COPY statement instead; not all that certain what the best way to do this with mysql is.  LOAD DATA LOCAL INFILE might be your best bet?  Otherwise, I'd consult with springle who is our resident mysql magician.  :-)
[19:22:00] <a930913>	 Coren: The wha? I'm using a series of INSERTs, is that not right?
[19:22:42] <Coren>	 a930913: It's /correct/, insofar as it will do the job, but I'm pretty sure that you can do a mass insert with LOAD DATA LOCAL INFILE in a way that's much more efficient.
[19:23:16] <Coren>	 a930913: I'm pretty sure that your task is currently dominated by the overhead of the individual insert statements.
[19:25:12] <Coren>	 http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html might help, too
[19:27:19] <Coren>	 A cursory reading of the related http://dev.mysql.com/doc/refman/5.5/en/insert-speed.html also suggest that LOAD DATA is up to 20x faster than inserts, too.
[19:27:41] <a930913>	 Yeah, just looking at the MariaDB version of that.
[19:28:10] <a930913>	 Coren: Do I use the filename /data/project/?
[19:29:30] <Coren>	 If you use LOAD DATA LOCAL INFILE then you need the filename as seen by the client - i.e.: with the path name you'd use to look at it.
[19:30:59] <a930913>	 Coren: Yeah, but if the db server can read it direct?
[19:31:13] <a930913>	 Is it accessible from the db server?
[19:32:05] <Coren>	 a930913: No; the db server lives on a separate network; and runs as a user that wouldn't have the right permissions anyways.
[19:32:29] <a930913>	 Ok.
[19:32:41] <a930913>	 Coren: Do you know the format for this INFILE?
[19:34:03] <Coren>	 You get to pick it.  The defaults, IIRC, are tab-separated columns, with newlines between the rows.
[19:35:04] <Coren>	 look at https://mariadb.com/kb/en/mariadb/documentation/sql-commands/data-manipulation/load-data-into-tables-or-index/load-data-infile/
[19:36:05] <a930913>	 Hmm, "SHOW PROCESSLIST" suggests that the bottleneck might be where it searches to see if a piece of data can be normalised...
[19:37:07] <nosy>	 hey
[19:37:13] <nosy>	 d like to file a bug for the osm wmflabs hosting
[19:37:17] <nosy>	 which category should i choose in bugzilla?
[19:42:00] <Coren>	 nosy: It doesn't look like the OSM people have created a component for their work.  :-)
[19:42:09] <Coren>	 nosy: You should probably poke them to do so.
[19:42:22] <nosy>	 who'd that be?
[19:42:31] <nosy>	 i mean who should i poke? ;)
[19:42:51] <nosy>	 thought there was some osm integration team forming
[19:46:16] <wikibugs>	 3Wikimedia Labs / 3tools: Hosting OSM tiles via HTTPS - 10https://bugzilla.wikimedia.org/70680 (10nosy) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier HTTPs delivery for OSM tiles currently results in certificate problems.  If URLs like https://c.tiles.wmflabs.org/hikebike/4/8/4.png are used to delive...
[19:46:17] <Coren>	 nosy: I honestly don't know who leads this.  Alexandros is the contact on our side; he probably will be able to tell you.
[19:46:24] <nosy>	 Coren: filed a toollabs ticket for now
[19:46:31] <Coren>	 nosy: That works.
[19:46:44] <wikibugs>	 3Wikimedia Labs: WMFLabs: Delete instance failed to "remove its DNS entry" - 10https://bugzilla.wikimedia.org/62770#c2 (10Andrew Bogott) Hm, maybe this is fixed by https://gerrit.wikimedia.org/r/#/c/119537/?
[19:54:14] <wikibugs>	 3Wikimedia Labs: WMFLabs: Delete instance failed to "remove its DNS entry" - 10https://bugzilla.wikimedia.org/62770#c3 (10Andrew Bogott) ...also https://gerrit.wikimedia.org/r/#/c/119698/1
[20:08:13] <wikibugs>	 3Wikimedia Labs: WMFLabs: Delete instance failed to "remove its DNS entry" - 10https://bugzilla.wikimedia.org/62770#c4 (10Andrew Bogott) This looks fixed!  It temporarily broke again because the jobqueue was broken.
[20:26:28] <wikibugs>	 3Wikimedia Labs / 3tools: Cannot start java processes using the grid engine - 10https://bugzilla.wikimedia.org/67588#c4 (10nosy)  tools.toolserverdb@tools-login:~$ sh -x ~/ns-updater/startUpdate.sh + which java + JAVA=/usr/bin/java + export _JAVA_OPTIONS=-Xmx150M + CONFIG=/data/project/toolserverdb/ns-update...
[20:50:30] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster api.php, index.php, load.php return 404 (caused failed browser tests) - 10https://bugzilla.wikimedia.org/70648#c2 (10Antoine "hashar" Musso) mw-api-siteinfo.py is in the repository integration/jenkins.git and should probably have better error handling w...
[21:08:50] <a930913>	 Coren: How can I make a bash script start many instances of the same script, but with different parameters?
[21:10:03] <Coren>	 a930913: I'm not sure what you're asking exactly.  Moar context?
[21:10:39] <a930913>	 Coren: Sorry, jsub.
[21:10:53] <a930913>	 Want many jobs with different parameters.
[21:12:58] <a930913>	 Coren: I.e. jsub parse.py $FILE, for FILE IN $FILES
[21:14:04] <Coren>	 a930913: A shell script could do it:  for foo in file1 file2 file3; do jsub blahblah.py $foo; done
[21:14:50] <a930913>	 Coren: I thought you couldn't submit arguments with jsub?
[21:14:59] <a930913>	 It could only be a pure executable.
[21:15:59] * a930913 wonders where he read that...
[21:16:57] <Coren>	 It has to be executable, but you can still give it arguments.  :-)  The only real difference with qsub is that you can't pipe a script into it.
[21:17:51] <a930913>	 Hmm. Anyway, DDoS submitted :p
[21:19:25] <Coren>	 It's not a DDOS, it's what gridengine is /for/  :-)
[21:19:58] <a930913>	 Oh ****.
[21:20:37] <a930913>	 I ran the old parser script...
[21:20:50] * a930913 introduces his face to the desk.
[21:21:44] <a930913>	 Good thing they all have the same name :) "qdel parser"
[21:21:48] <Coren>	 Palms hurt less and have mostly the same symbolic value.  :-)
[21:23:02] <a930913>	 For the first time, I'm piping qstat into less :o
[21:24:43] <a930913>	 Holy ****.
[21:25:09] <a930913>	 I've never really used the gridengine as a grid before.
[21:25:40] <a930913>	 It just went and parsed gigabytes of text, right before my eyes.
[21:25:46] <Coren>	 :-)
[21:26:21] <Coren>	 Moar parallel!
[21:26:28] <a930913>	 Coren: How much read can /data/project/ handle?
[21:27:13] <Coren>	 a930913: Taking around the various inneficiencies, and depending on the actual read pattern, probably somewhere around 5-6gbps
[21:27:56] <Coren>	 a930913: Though you'd be hard pressed to actually sustain that over a long period because of contention with other tools.
[21:28:12] <a930913>	 So it could load up all the grid in three seconds.
[21:28:26] <a930913>	 Then 10x parallel. :o
[21:28:40] <Coren>	 Sure, but it'd level off fairly quickly as the disk scheduler partitions the usage.
[21:28:54] <a930913>	 I really out to do these things more often.
[21:29:09] <Coren>	 bbl, dinner
[21:29:22] <a930913>	 Coren: Thanks btw :)
[21:33:26] <a930913>	 I'm using cat for what it was originally intended for :o
[21:34:35] <mutante>	 a930913: so you dont get this award http://www.smallo.ruhr.de/award.html
[22:19:14] <a930913>	 Has tools-db b0rked?
[23:39:50] <RoanKattouw>	 Hey labs opsen, could someone help me figure out what the hell happened to my instance deployment-parsoidcache01 in the deployment-prep project? It's not letting me SSH in, it appears to be unable to be rebooted, and its console log looks concerning
[23:41:57] <RoanKattouw>	 Coren: I know you're mostly a tool labs person, but do you know who I should ping about mysterious stuff like thiis?
[23:46:57] <chrismcmahon>	 RoanKattouw: jeremyb was hacking in that area today, beta labs melted down a bit from puppet misconf aiui
[23:47:08] <chrismcmahon>	 greg-g: ^^
[23:47:42] <RoanKattouw>	 Well this goes beyond beta labs itself, it seems to be an OpenStack problem