[00:26:09] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL 33.33% of data above the critical threshold [0.0] [01:06:10] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK Less than 1.00% above the threshold [0.0] [02:05:25] !log reading-web-staging Added Robmoen, Jhernandez and Jdlrobson as project admins [02:06:10] Morebots is AWOL [02:07:39] from this channel [02:07:52] right. [02:15:54] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL 25.00% of data above the critical threshold [0.0] [02:55:58] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK Less than 1.00% above the threshold [0.0] [06:55:28] 6Labs, 10Tool-Labs, 10Pywikibot-OAuth: Add OAuth to https://tools.wmflabs.org/pywikibot-testwiki/ - https://phabricator.wikimedia.org/T104291#1466327 (10VcamX) @Ladsgroup I think I'm in tools project now. Can you try to add me to tools.pywikibot-testwiki group again? Thank you! [09:01:55] 6Labs: Make an evacuation plan for labs instances - https://phabricator.wikimedia.org/T106144#1466419 (10yuvipanda) Should we make an OpenStack upgrade part of the goal? [09:03:39] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466427 (10Magnus) 3NEW [09:59:42] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466503 (10jcrespo) @Magnus, You are limiting yourself by reading and writing to a table engine that does not support those concurrent operations (MyISAM). "`Waiting for table level lock`" Aside from th... [10:08:12] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466507 (10Magnus) Actually, I converted from InnoDB to MyISAM a while ago because the reads on InnoDB were so slow the tool (mix'n'match) would stop working. Apparently, I have only bought some time...... [10:21:06] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466520 (10jcrespo) @Magnus, in MyISAM, reads block writes (and I do not think concurrent inserts will work), and you have at least **4 reads connected all the time** plus an import: ``` thread1: SELECT... [10:21:22] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466521 (10Magnus) To be precise, I had 703 row updates on that database yesterday. If THAT is breaking the server, the server was already broken to begin with. [10:25:52] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466531 (10Magnus) Just saw your previous reply. Not sure where the "catalog=2" queries come from. Restarting the webservice in case it's some runaway web script. So, are you saying I can "just" connect... [10:30:55] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466536 (10Magnus) show full processlist; 3820629 p50380g50851 10.68.17.228:60067 p50380g50851__mixnmatch_p Query 0 init show full processlist 0.000 3820637 p50380g50851 10.68.17.228:60131 NULL Sleep 17... [10:33:51] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466538 (10jcrespo) @Magnus, there is *someone* accessing your tables, creating problems for you and everyone else. Regarding other database hosts, I am not proficient on labs administration (only a DBA... [10:34:05] 10Tool-Labs-tools-Other: Restore connectivity project tools - https://phabricator.wikimedia.org/T106373#1466542 (10Nemo_bis) 3NEW [10:37:34] 10Tool-Labs-tools-Other: Restore connectivity project tools - https://phabricator.wikimedia.org/T106373#1466557 (10Nemo_bis) [11:02:17] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466592 (10jcrespo) @Magnus, willing the other connections seems to have an impact, stopping the swapping and probably improving your import performance (it may take a while, however). Regarding other da... [11:49:14] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1466611 (10Magnus) YES! Now it's fast again! Thanks, and whatever you did, keep doing it :-) [11:54:14] 6Labs, 7Mobile, 3Mobile-App-Sprint-62-Android-Summer-Breeze, 3Wikipedia-Android-App: Decide what to do with the android-build machine - https://phabricator.wikimedia.org/T104190#1466626 (10Dbrant) [14:18:14] (03PS1) 10Sitic: Cleanup backend code [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226076 [14:20:37] (03PS2) 10Sitic: Cleanup backend code [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226076 [14:20:53] (03CR) 10Sitic: [C: 032 V: 032] Cleanup backend code [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226076 (owner: 10Sitic) [14:44:09] (03PS1) 10Sitic: Only use one redis channel per client [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226082 [14:44:43] (03CR) 10Sitic: [C: 032 V: 032] Only use one redis channel per client [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/226082 (owner: 10Sitic) [15:52:10] 6Labs, 6Phabricator, 5Patch-For-Review, 7Puppet: On labs phabricator references security extension even though it isn't present - https://phabricator.wikimedia.org/T104904#1467102 (10chasemp) 5Open>3Resolved [15:52:44] 6Labs, 10Beta-Cluster, 6operations, 7Monitoring: Setup (simple) catchpoint monitoring and metrics for enwiki betacluster just like production - https://phabricator.wikimedia.org/T97865#1467105 (10greg) [17:24:34] 6Labs, 6Discovery, 10Maps: WikiMiniAtlas (wma.wmflabs.org) is still down - https://phabricator.wikimedia.org/T104417#1467691 (10dschwen) 5Open>3Resolved a:3dschwen Muchas gracias to Coren, who helped me unf$%k my WMA instance (which was unsshloginable :-) [17:46:20] 6Labs, 10Labs-Infrastructure: MySQL spectacularly slow on Labs - https://phabricator.wikimedia.org/T106372#1467831 (10jcrespo) So, the actual recommendation is to use `tools.db` host, with the same user and password you are using now if your application does not have a hard requirement on reading from the repl... [19:09:30] 6Labs, 6Phabricator, 5Patch-For-Review, 7Puppet: On labs phabricator references security extension even though it isn't present - https://phabricator.wikimedia.org/T104904#1468296 (10mmodell) [19:18:06] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1468323 (10coren) @yuvipanda: It's keeping the filesystem reasonably small (and operations on them more parallelizable) by spliting along project lines, yes. So right now we've spun off `tools` and `maps` with everythin... [20:03:51] 10Quarry: 'New query' highlighted when looking at existing queries - https://phabricator.wikimedia.org/T106411#1468621 (10yuvipanda) 3NEW [20:26:38] mw-extension is the old mediawiki_singlenode role... [20:26:45] not sure how that showed up on labs-vagrant? [20:26:55] me neither... [20:27:02] so instance is mf-browser-tests on https://wikitech.wikimedia.org/wiki/Special:NovaInstance [20:27:08] under mobile-smoketests [20:27:25] looking [20:28:29] !log mobile-smoketests Added myself (BryanDavis) as a project admin [20:28:43] YuviPanda: labs-morebots down? [20:28:50] it could be. it often is. [20:29:54] jdlrobson: you have "role::mediawiki-install::labs" enabled. You don't want that [20:30:02] Just role::labs::vagrant [20:30:07] which you also have enabled [20:31:10] i just copied the gather one.. which has both [20:31:34] ok so i removed it [20:31:52] does sudo puppet agent -tv work now? [20:31:58] yup [20:32:07] * jdlrobson feels stupid [20:32:14] meh [20:32:25] these rolls are a bit of a mess [20:32:32] roles even [20:32:39] bd808: we should clean up the global ones [20:32:46] and get rid of mediawiki_singlenode [20:32:55] YuviPanda: yes. I got rid of a small amount of crap last week [20:33:04] <3 [20:33:06] but it could use a good audit [20:33:20] the deployment-bastion project was a wreck [20:33:23] yeah [20:34:05] bd808: I'm going to do a block of deletions now if they have no instances (as seen by https://tools.wmflabs.org/watroles/role/db::core) [20:34:10] i'm provisioning vagrant as we speak [20:34:15] 25 hosts have role::mediawiki-install::labs enabled :/ [20:34:25] i canhaz a big green button to do all this in future? :) [20:35:46] the button shall be green, not yellow, not red, not purple, but green. It shall neither be light of hue, nor dark of hue, but simple green. [20:36:03] ni [20:36:24] jdlrobson: yes, if we can hire a few more people and/or give me / bd808 more copious free time :( [20:36:27] * bd808 sends chasemp to find a shrubbery [20:36:36] the holy vagrant deployment [20:36:38] heh [20:37:15] My holy grail would be `vagrant up --provider wmflabs` [20:37:24] sudo: /usr/local/bin/run-git-update: command not found < [20:37:30] from running `labs-vagrant git-update` [20:37:44] jdlrobson: hmmmm.... [20:37:46] chasemp: see, even you had added phabricator roles to global pool than the project specific pool [20:37:57] provision had a few errors and warnings [20:38:20] YuviPanda: do you mean it's phab::labs and not labs::phab or some such? [20:38:36] jdlrobson: logging in to look around [20:38:52] chasemp: no, I mean in the left sidebar on wikitech, under 'manage puppet groups' it should be added just to the phabricator project than to all of labs. Not sure if that was you, though [20:39:01] so the checkbox now shows up on all projects instead of just the phabricator project [20:39:14] i seem to have a vagrant installed but nothing at /srv/mediawiki [20:39:17] I don't think I did that actually as I wasn't sure how it was done at the time [20:39:31] jdlrobson: You're going to be mad at me. [20:39:39] labs-vagrant doesn't work on jessie [20:39:53] you need to build a trusty host [20:40:06] but also, there are more projects using it than just phabricator so I guess it worked out [20:40:10] or were even [20:41:10] jdlrobson: the labs-vagrant managed MW install would be at /srv/vagrant/mediawiki [20:41:17] but it's not gonna work here [20:41:37] because the puppet rules in mw-vagrant don't know how to work on jessie [20:41:58] chasemp: no, only chastest and phab projects are using that https://tools.wmflabs.org/watroles/role/role::phabricator::labs [20:42:13] I'm not too happy that the default labs instance type is jessie honestly [20:42:41] ah the momentum faded, the cleaner for greater namespace approach would be to only make this role avail to these two projects you are saying? [20:43:12] chasemp: yes! :) [20:43:26] bd808: I'm doing a cleanup of the global space crap [20:43:27] at the time many ppl in serach of a feature request were shown the way to developing it themselves but that zest seems to have subsided [20:43:32] mm so.. i'm confused.. what did i do wrong? [20:43:55] and isn't gather-browser-tests on jessie.. how did i set that up without issues? [20:44:18] It' must not really be using labs-vagrant [20:44:54] and instead using role::mediawiki-install::labs and an hand build MW config [20:45:40] Since all the prod MW servers are trusty that's really what you should use for MW stuff in labs [20:46:02] jessie is the new hotness for misc and services hosts in prod [20:46:24] but not MW servers, job runners or image and video scalers [20:48:49] 6Labs, 10Tool-Labs: new labs host sends out "mpt raid status change" emails - https://phabricator.wikimedia.org/T104779#1468808 (10scfc) The issue is occurring for `tools-web-static-01` and `tools-web-static-02` now as well, and I don't quite understand it. https://gerrit.wikimedia.org/r/#/c/215994/ ("base: k... [20:52:37] 6Labs, 10Tool-Labs: new labs host sends out "mpt raid status change" emails - https://phabricator.wikimedia.org/T104779#1468827 (10scfc) On second look: `modules/labs_vmbuilder/templates/vmbuilder.cfg.erb` lists `mpt-status` under `addpkg`. Could it be that thus `mpt-status` is installed before Puppet install... [20:54:20] 6Labs, 10Tool-Labs: new labs host sends out "mpt raid status change" emails - https://phabricator.wikimedia.org/T104779#1468843 (10scfc) (The package was added to the `addpkg` list by https://gerrit.wikimedia.org/r/#/c/217667/ ("Add many more packages to base image").) [21:07:31] 6Labs, 10Beta-Cluster, 6operations, 7Monitoring: Setup (simple) catchpoint monitoring and metrics for enwiki betacluster just like production - https://phabricator.wikimedia.org/T97865#1468877 (10greg) We talked about this on ops list: https://lists.wikimedia.org/mailman/private/ops/2015-July/049244.html... [21:39:28] bd808: is there an easy way to set vagrant to a specific MW release? [21:40:02] I'm thinking about how to turn https://phabricator.wikimedia.org/T65282#1466587 into something reproducible [21:45:53] 6Labs: create labs project for barry the browser bot - https://phabricator.wikimedia.org/T106429#1469007 (10bmansurov) 3NEW [21:46:12] 6Labs: create labs project for barry the browser bot - https://phabricator.wikimedia.org/T106429#1469017 (10bmansurov) [21:46:14] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1469016 (10bmansurov) [21:48:20] bd808: we don't have new project guidelines yet but I'm wondering what to do with https://phabricator.wikimedia.org/T106429#1469007 [21:48:57] hmm, I wonder if they know you can have multiple instances in a project? I saw quite a bit of nomenclature confusion [21:48:58] let me ask [21:49:18] but you said projects had to be granular! ;) [21:49:39] bd808: indeed, so I'm inclined to grant this provided they do know it's possible to have multiple instances in a project :) [21:50:56] tgr: I think going back to <1.23 is possibly tricky, but you control the vagrant/mediawiki git clone so if you want an old branch you can just check it out [21:51:35] There is also some undocumented magic that we created for the fundraising folks that you can use to setup a second MW checkout [21:51:55] 6Labs: create labs project for barry the browser bot - https://phabricator.wikimedia.org/T106429#1469034 (10yuvipanda) Hi! Do you just need an instance to test the instructions? Isn't there already a project that @rmoen used for the exact purpose? You can create multiple instances under one project. I'm unclear... [21:53:04] tgr: if you want to get that fancy, take a look at https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/payments/manifests/init.pp [21:54:24] 6Labs: create labs project for barry the browser bot - https://phabricator.wikimedia.org/T106429#1469041 (10bmansurov) Thanks for the response, @yuvipanda. The purpose is to test the instructions listed at [1]. I want to make sure each step is reproducible. [1] https://www.mediawiki.org/wiki/Reading/Setting_up_... [21:56:42] 6Labs: create labs project for barry the browser bot - https://phabricator.wikimedia.org/T106429#1469043 (10yuvipanda) I will say that point two there is wrong - you should only request a new project if there isn't an existing project that will fit this. Assuming this is being done for the mobile team, why not i... [21:56:50] bd808: ^ do weigh in if you feel like [21:56:50] bd808: I was just wondering if there was a role or vagrant command to control that [21:57:13] not right now, no [21:57:19] tgr: ^ [21:57:40] using git seems messy, the DB is not rolled back, you need to figure out how to deal with composer in that version etc [21:57:53] agreed [21:58:06] but this hasn't been a common use case [22:27:05] 6Labs: create labs project for barry the browser bot - https://phabricator.wikimedia.org/T106429#1469146 (10bmansurov) OK, thanks. I'll close this ticket as resolved. The instructions need to be updated to use an existing project. [22:27:37] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1469149 (10bmansurov) [22:27:37] 6Labs: create labs project for barry the browser bot - https://phabricator.wikimedia.org/T106429#1469147 (10bmansurov) 5Open>3Resolved a:3bmansurov [22:45:24] 6Labs, 7Tracking: Sn1per mediawiki testing labs project - https://phabricator.wikimedia.org/T106086#1469200 (10yuvipanda) Hmm, we prefer not providing projects that are 'per-person' but projects that are 'per-project'. If you tell us what project you are primarily working on, we could probably find you a proje... [22:53:24] 6Labs, 3Labs-Sprint-104, 3Labs-Sprint-105, 7Puppet: Allow per-host hiera overrides via wikitech - https://phabricator.wikimedia.org/T104202#1469226 (10yuvipanda) [22:53:26] 6Labs: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1469225 (10yuvipanda) [23:02:03] 6Labs: failing puppet - https://phabricator.wikimedia.org/T106442#1469242 (10bmansurov) 3NEW [23:03:22] 6Labs, 10Labs-Vagrant: failing puppet - https://phabricator.wikimedia.org/T106442#1469265 (10yuvipanda) [23:26:20] bd808: the global namespace is much much cleaner now. Lots of unused shit. [23:26:40] bd808: Ideally I think only the self hosted puppetmaster and labs-vagrant should be there. and /srv too, probably. [23:26:57] hmm, probably the LAMP one too [23:32:05] so...I'm seeing some new instances come up in the staging project where the realm doesn't seem to be getting set in ldap. [23:33:20] whoops [23:33:22] thcipriani: instance name? [23:34:03] YuviPanda: staging-wdqs-test.staging.eqiad.wmflabs [23:34:48] hmm [23:34:48] https://tools.wmflabs.org/watroles/variable/instancename/staging-wdqs-test [23:34:50] you're right [23:35:08] SMalyshev mentioned that one was acting funny, I said it was probably a fluke so they spun up staging-wdqs2 and said they got the same error [23:35:23] thcipriani: how long ago was this? [23:35:32] just today [23:35:40] like, 4-8h ago or just now? [23:35:49] wdqs2 was just now [23:35:59] ok, let me see if I was the one who borked it [23:36:03] not sure on staging-wdqs-test, less than 4 hours I would guess [23:36:10] probably more like last hour [23:37:03] 6Labs, 7Tracking: Sn1per mediawiki testing labs project - https://phabricator.wikimedia.org/T106086#1469400 (10Sn1per) >>! In T106086#1469200, @yuvipanda wrote: > Hmm, we prefer not providing projects that are 'per-person' but projects that are 'per-project'. If you tell us what project you are primarily worki... [23:38:42] thcipriani: can you recreate now? [23:39:33] * thcipriani tries [23:40:08] YuviPanda: lgtm: https://tools.wmflabs.org/watroles/variable/instancename/staging-wdqs-test2 [23:40:15] thcipriani: sweet [23:40:21] ^ I like that tool, very fancy [23:40:46] much nicer than my long ldapsearch copy and paste thing :) [23:41:14] thcipriani: :D wrote it a couple of weeks ago, needs a home page with a few text boxes