[00:00:30] marxarelli: if it's not for production though, tool labs provides a nice cdnjs mirror you can use. [00:00:57] https://tools.wmflabs.org/cdnjs/ [00:01:29] Has its own tools-static server [00:08:04] (03PS1) 10Dduvall: Use tool labs CDN for dependencies [integration/raita] - 10https://gerrit.wikimedia.org/r/208045 [00:08:25] RECOVERY - Puppet failure on integration-slave-trusty-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [00:09:03] Krinkle: ^ using tools cdn for now :) [00:11:05] cool [00:16:48] (03PS2) 10Dduvall: Use tool labs CDN for dependencies [integration/raita] - 10https://gerrit.wikimedia.org/r/208045 [00:18:19] (03CR) 10Dduvall: [C: 032] Use tool labs CDN for dependencies [integration/raita] - 10https://gerrit.wikimedia.org/r/208045 (owner: 10Dduvall) [00:19:54] (03Merged) 10jenkins-bot: Use tool labs CDN for dependencies [integration/raita] - 10https://gerrit.wikimedia.org/r/208045 (owner: 10Dduvall) [02:07:12] PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:29:18] PROBLEM - Puppet staleness on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [04:03:22] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [04:18:22] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.027 second response time [04:39:54] Yippee, build fixed! [04:39:55] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #430: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/430/ [04:49:38] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<11.11%) [05:34:25] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #405: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/405/ [05:40:41] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #56: FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/56/ [06:19:38] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<55.56%) [06:39:39] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:58:13] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #427: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/427/ [08:21:10] 6Release-Engineering, 3Team-Practices-This-Week: Test phabricator sprint extension updates - https://phabricator.wikimedia.org/T95469#1251491 (10mmodell) @Christopher: go ahead and merge + tag. I will try to get this published on Wednesday. [08:53:28] Yippee, build fixed! [08:53:28] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #586: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/586/ [10:50:16] 10Deployment-Systems, 6operations, 7Graphite: [scap] Deploy events aren't showing up in graphite/gdash - https://phabricator.wikimedia.org/T64667#1251557 (10fgiunchedi) this should be working now in graphite at least https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1430476877.683&target=dra... [13:06:06] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #633: STILL FAILING in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/633/ [13:42:56] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1251822 (10mobrovac) I would really prefer if the structure could mimic production, where all of our node.js services reside on the SCA cluster. To that end, in the Beta Cluster we already have `deployment-sc... [14:16:31] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1251858 (10Yurik) @mobrovac, sure, reusing the same VM is fine with me. @akosiaris said he is willing to help with deploying it? [14:35:20] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1251877 (10mobrovac) Cool. I added the `role::graphoid` Puppet class to `deployment-sca01`. Note that graphoid's config for deployment-prep is [already in Hiera](https://phabricator.wikimedia.org/diffusion/OP... [14:36:42] Yippee, build fixed! [14:36:42] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #105: FIXED in 8 min 40 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/105/ [14:48:26] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:07:44] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1251946 (10mobrovac) Graphoid is now deployed: ``` mobrovac@deployment-sca01:~$ curl localhost:19000/_info {"name":"graphoid","version":"0.1.1","description":"renders vega graphs from mediawiki pages","home"... [15:12:38] 10Beta-Cluster: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1251957 (10mobrovac) 3NEW [15:16:21] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1251977 (10greg) Yo @chasemp, is this fallout from that change from yesterday? [15:18:28] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:32:05] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1252019 (10chasemp) >>! In T97775#1251977, @greg wrote: > Yo @chasemp, is this fallout from that change from yesterday? This would be an issue with trebuchet umask I imagine so... [15:41:09] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1252055 (10mobrovac) >>! In T97775#1252019, @chasemp wrote: > If anything this should really be `sudo chown -R trebuchet:deployment` Current practice seems to disagree on that... [15:45:26] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1252075 (10chasemp) sure, I mean all of those should be owned by trebuchet and deployment since deployment is the group for deployers. wikidev is the default group for all user... [15:51:09] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1252101 (10thcipriani) There are a couple of approaches out there to solve this problem. Check out: https://gerrit.wikimedia.org/r/#/c/201344/2 I have another patch that is a l... [15:51:34] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1252102 (10Yurik) @mobrovac, awesome!!! We need to expose it as graphoid.wikimedia.beta.wmflabs.org [16:05:06] 10Beta-Cluster, 6operations, 7Puppet: Trebuchet on deployment-bastion: wrong group owner - https://phabricator.wikimedia.org/T97775#1252131 (10chasemp) >>! In T97775#1252101, @thcipriani wrote: > There are a couple of approaches out there to solve this problem. Check out: https://gerrit.wikimedia.org/r/#/c/2... [16:41:20] 6Release-Engineering, 10Wikipedia-iOS-App, 3Mobile-App-Sprint-56-iOS: Sprint 55 Release - https://phabricator.wikimedia.org/T95969#1205258 (10BGerstle-WMF) [16:51:46] (03PS1) 10Dduvall: Remove support for MEDIAWIKI_PASSWORD_VARIABLE [selenium] - 10https://gerrit.wikimedia.org/r/208119 [16:55:23] 10Deployment-Systems, 6operations, 7Graphite, 5Patch-For-Review: [scap] Deploy events aren't showing up in graphite/gdash - https://phabricator.wikimedia.org/T64667#1252284 (10bd808) >>! In T64667#1251557, @fgiunchedi wrote: > > @bd808 is scap sending statsd counters? if so the names will change upon next... [17:09:32] 10Deployment-Systems, 6operations, 7Graphite, 5Patch-For-Review: [scap] Deploy events aren't showing up in graphite/gdash - https://phabricator.wikimedia.org/T64667#1252345 (10fgiunchedi) >>! In T64667#1252284, @bd808 wrote: >>>! In T64667#1251557, @fgiunchedi wrote: >> >> @bd808 is scap sending statsd co... [17:40:41] 6Release-Engineering, 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Grant access for aklapper to phab-admins - https://phabricator.wikimedia.org/T97642#1252553 (10Dzahn) p:5Triage>3Normal [17:57:07] 10Beta-Cluster: Special:GlobalRenameRequest throws a "Cannot access the database" error - https://phabricator.wikimedia.org/T97813#1252615 (10Glaisher) 3NEW [17:59:39] 10Beta-Cluster, 10SUL-Finalization: Special:GlobalRenameRequest throws a "Cannot access the database" error - https://phabricator.wikimedia.org/T97813#1252649 (10greg) [18:09:43] Project beta-scap-eqiad build #51140: FAILURE in 5.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51140/ [18:09:46] bd808: Hi, thanks for the puppet tip! I don't know if that will work in this case, cos 0640 permissions for example will prevent the www-data user from reading the settings files. [18:13:34] awight: agreed. strict permissions are generally counter productive in mw-vagrant. It's tuned to be a development environment. [18:16:13] ok, makes sense! [18:16:51] I'd like to reuse these modules for labs installation one day, but I can deal with that later. [18:17:51] You can move the settings into params on the class and then set them via hiera for easier reuse [18:19:08] I'm ... not 100% sure that setting a hiera value == a factor fact is easy. Might have to read up on that [18:19:13] 10Beta-Cluster, 10SUL-Finalization, 5Patch-For-Review: Special:GlobalRenameRequest throws a "Cannot access the database" error - https://phabricator.wikimedia.org/T97813#1252726 (10thcipriani) Hmm, looks like dawiki needs to be removed from wikiversions-labs.json as well (see: https://integration.wikimedia.o... [18:27:13] 10Beta-Cluster, 7Blocked-on-RelEng, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, and 3 others: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1252793 (10Legoktm) So....right now CentralAuth is broken because these wikis hav... [18:44:56] <^d> twentyafterfour: I think addWiki can take the new wiki's dbname as it's --wiki too, more I'm looking at it [18:45:10] <^d> Long as you've already rebuild the cdb. [18:45:29] <^d> Wait, addWiki is supposed to do that for you [18:45:30] <^d> wtf... [18:45:38] <^d> you do need a real wiki, and step (1) is wrong [18:45:53] new wiki === black magic [18:46:05] <^d> And /actually/ you need an existing wiki in the same cluster as where you want to make it [18:46:07] * bd808 sacrifices a goat [18:46:15] <^d> Presumably this is s3, but if you pick --enwiki it'll be wrong [18:46:40] <^d> addWiki sucks. [18:46:44] <^d> I think I'm going to rewrite it [18:51:35] 10Beta-Cluster, 7Blocked-on-RelEng, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, and 3 others: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1253021 (10Legoktm) Reverted the all-labs.dblist/wikiversions.json changes in htt... [18:58:07] how do I manually do a sync-file in beta? [18:58:25] unknown environment `default` (MediawikiSelenium::ConfigurationError) [18:58:30] Anyone recognize that? [18:58:42] http://fpaste.org/217634/6717143/raw/ [18:59:01] marktraceur: yep. what repo? [18:59:41] marxarelli: UploadWizard [19:00:09] marktraceur: mw-selenium (>= 1.0) throws that error if MEDIAWIKI_ENVIRONMENT isn't set and no entry for 'default' exists in environments.yml [19:00:23] Uh...hm [19:00:25] PROBLEM - Free space - all mounts on deployment-eventlogging02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging02.diskspace._var.byte_percentfree (<100.00%) [19:00:29] What am I supposed to do about that? :) [19:00:44] marktraceur: are you seeing that error in the ci build or locally? [19:00:48] Locally. [19:01:38] marktraceur: ah, looking now [19:02:56] 10Beta-Cluster, 10SUL-Finalization, 5Patch-For-Review: Special:GlobalRenameRequest throws a "Cannot access the database" error - https://phabricator.wikimedia.org/T97813#1253110 (10Legoktm) 5Open>3Resolved a:3Legoktm http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:GlobalRenameRequest works now. [19:03:44] legoktm: looks like everything is getting synced out via jenkins currently: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51145/console [19:03:56] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: CentralAuth on beta throws "Cannot access the database: Unknown database 'dawiki'" for new users - https://phabricator.wikimedia.org/T97388#1253113 (10Legoktm) [19:03:58] 10Beta-Cluster, 10SUL-Finalization, 5Patch-For-Review: Special:GlobalRenameRequest throws a "Cannot access the database" error - https://phabricator.wikimedia.org/T97813#1253114 (10Legoktm) [19:03:59] marktraceur: so, it looks like there's no default entry in `environments.yml`. the newest version of mw-selenium uses 'mw-vagrant-host' as the default using a yaml anchor [19:04:05] marktraceur: http://git.wikimedia.org/blob/mediawiki%2Fselenium.git/HEAD/templates%2Ftests%2Fbrowser%2Fenvironments.yml [19:04:13] thcipriani: I got a little impatient and synced it out myself :P [19:07:40] marktraceur: tl;dr: just create a `default:` entry in environments.yml for the configuration you want mw-selenium to use by default [19:08:11] marktraceur: when you want to run against beta, you can do `MEDIAWIKI_ENVIRONMENT=beta bundle exec cucumber ...` [19:12:08] Yippee, build fixed! [19:12:08] Project beta-scap-eqiad build #51146: FIXED in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51146/ [19:12:49] marxarelli: Should I put that entry into the git repo, or just do it locally and gitignore or something? [19:14:33] thcipriani: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51145/console weird? [19:14:45] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/51146/console was fine though [19:15:24] looks like it was just a problem with the lock file missing in 51145 [19:16:23] o.O [19:16:39] legoktm: sync-file was also complaining about the lock file when you tried to manually sync [19:16:51] that was a permission error though? [19:16:53] my guess is there was some weird interaction there... [19:17:04] maybe I clobbered the lock file when running it manually? [19:17:13] though the lock should have prevented that :P [19:17:20] heh, you'd think [19:18:15] I wonder how much exercise the lock file mechanism gets: probably not too much. Maybe there is a bug therein. [19:18:27] marxarelli: Because really, it seems silly to put my passwords for my local testing wiki into the UW git repo. [19:18:41] Or, for that matter, the passwords for testing users on testwiki and betawiki. [19:22:32] 10Beta-Cluster, 7Blocked-on-RelEng, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, and 3 others: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1253146 (10demon) >>! In T90683#1253021, @Legoktm wrote: > The databases need to... [19:23:34] <^d> I had lots to say about creating a wiki ^ [19:25:04] Yeah I notice the aawiki/s3 thing yesterday when looking at addWiki [19:25:11] noticed* [19:25:20] <^d> In theory aawiki is mostly always ok [19:25:26] <^d> And for beta it will be since we don't cluster [19:25:40] tl;dr: stop creating new wikis ;) [19:25:47] <^d> CLEARLY [19:25:49] <^d> NO MORE WIKIS [19:26:14] PROBLEM - Puppet staleness on integration-saltmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [19:26:36] <^d> lunchhhhh [19:34:32] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:56:17] marktraceur: right. passwords for shared or otherwise sensitive environments are left out. you can set those via environment variables [19:59:30] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [20:05:25] marktraceur: i was trying to simplify setup of browser tests with that feature so let me know if you see ways to make it better [20:05:56] e.g. have it look for a "local" config that can be ignored, etc. [20:06:23] maybe a `credentials.yml` file or something... iono [20:12:07] marxarelli: But I have the creds set in environment variables and it's still failing [20:12:56] marxarelli: And the passwords for test/beta are in the file as far as I can tell... [20:14:49] marktraceur: re: the passwords, i don't see them there (just fetched master) [20:14:58] Huh. [20:15:07] Oh, right. [20:15:10] Just the usernames. OK. [20:15:44] it may still fail if there's no `default` entry though, so you might just want to alias it to mw-vagrant-host [20:51:56] FrozenFire: you can build big ones, but they arn't as fun. they are just arial photography igs [20:52:21] ^^ ignore that .. my lovely kid is typing for me :) [20:52:27] or well, pushing buttons... [21:02:15] https://image-store.slidesharecdn.com/28ecb80d-ac34-430a-801a-6496db4b28cf-original.jpeg [21:02:17] >.> [21:09:27] quiddity: anything I should be worried about? :) [21:09:49] nope, just a "found on reddit", and this seemed the most appropriate channel to drop it in. :) [21:11:10] :) [21:42:34] PROBLEM - Puppet failure on integration-raita is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:49:41] ^ that's me [21:53:22] Are there already any MW-Vagrant Puppet modules which import wiki data? I'd like to customize the main page for paymentswiki... [21:54:15] awight: multimediaviewer [21:54:21] marxarelli: ok, thanks! [21:54:39] Awesome, I see it. [21:55:00] awight: actually, sentry might be a better example [21:55:05] k [21:55:55] marxarelli: I don't actually see where sentry imports anything. [21:56:14] ah, got it. role/files for some reason [21:56:16] awight: it uses mediawiki::import_text [21:56:29] rad [21:56:32] which, yeah, references a file containing the wikitext [21:56:47] I think that file should be moved into the sentry module... [21:57:56] it's in puppet/modules/role/files/sentry which i believe is the right place for it [21:58:35] consistent with the manifest/template organization at least [21:58:41] I trust you! Ruby makes me get a bit fetishistic about compartmentalizing stuff [21:58:59] I would have gone with puppet/modules/sentry/files [21:59:27] i would put it there if it were core to sentry itself [21:59:50] but it's more related to the role, so i think where's it's at now makes more sense [22:00:02] s/'s// [22:00:58] awight: .pp files ain't ruby btw :P it's its own beast! [22:01:07] It's so horrible... [22:01:11] haha [22:01:16] pseudodeclarative mush [22:01:31] it's a little 'funny' [22:14:43] (03PS1) 10Dduvall: Add pre-compiled tags to obviate node dependencies in production [integration/raita] - 10https://gerrit.wikimedia.org/r/208286 [22:15:34] (03PS2) 10Dduvall: Commit build/tags.js to obviate node dependencies in production [integration/raita] - 10https://gerrit.wikimedia.org/r/208286 [22:16:37] (03CR) 10Dduvall: [C: 032] Commit build/tags.js to obviate node dependencies in production [integration/raita] - 10https://gerrit.wikimedia.org/r/208286 (owner: 10Dduvall) [22:16:44] (03Merged) 10jenkins-bot: Commit build/tags.js to obviate node dependencies in production [integration/raita] - 10https://gerrit.wikimedia.org/r/208286 (owner: 10Dduvall) [22:17:38] 10Beta-Cluster, 6Labs, 7Monitoring: Setup (simple) catchpoint monitoring for betacluster - https://phabricator.wikimedia.org/T97865#1253670 (10yuvipanda) 3NEW [22:22:11] 10Beta-Cluster, 6Labs, 7Monitoring: Setup (simple) catchpoint monitoring for betacluster - https://phabricator.wikimedia.org/T97865#1253678 (10greg) [22:27:20] Does anyone know how to increase the browser test timeout locally? [22:27:29] I tried: [22:27:38] export PAGE_WAIT_TIMEOUT=10000; export ELEMENT_WAIT_TIMEOUT=10000; time bundle exec cucumber tests/browser/features/suppress.feature [22:28:14] but it still times out (https://phabricator.wikimedia.org/P597), and the overall run only takes 2.5 minutes. [22:32:36] RECOVERY - Puppet failure on integration-raita is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:25] matt_flaschen: it's hard to tell exactly what is timing out but it's probably not the page-object wait timeout [22:41:43] you wouldn't get a Net::ReadTimeout from that [22:42:00] it could be the webdriver connection possibly [22:42:33] matt_flaschen: run cucumber with `--backtrace` to get more info [22:42:50] I think the page is just slow to load, but I guess that's not the right timeout variable for that. [22:52:00] 10Beta-Cluster, 6Labs, 7Monitoring: Setup (simple) catchpoint monitoring for betacluster - https://phabricator.wikimedia.org/T97865#1253746 (10greg) See also {T88705} [22:57:08] marxarelli: Backtrace at https://phabricator.wikimedia.org/P598 [22:59:06] matt_flaschen: yeah, that looks like a timeout of the webdriver client's http connection [22:59:43] matt_flaschen: what version of mediawiki_selenium are you using? [23:00:13] marxarelli, 0.4.2 [23:00:38] darn. it would be super simple to override in 1.x [23:00:39] marxarelli, if I upgrade to latest, do you think it will break stuff? [23:00:46] oh yes :) [23:01:07] matt_flaschen: but there's a decent guide here https://doc.wikimedia.org/rubygems/mediawiki-selenium/file.UPGRADE.html [23:01:12] Cool, thanks. [23:01:49] marxarelli, that's not bad at all if that pretty much covers it. [23:02:54] matt_flaschen: zeljkof and i have followed it a few times already so unless you're doing something exotic it should cover you [23:03:49] matt_flaschen: oh, hold the phone! [23:04:03] BROWSER_TIMEOUT is already supported by mw-selenium [23:04:10] try that first [23:04:33] Oh, that does work? [23:06:38] matt_flaschen: looks like BROWSER_TIMEOUT with firefox is supported in 0.4.2 [23:06:49] 1.x should support it for all browsers [23:07:04] http://git.wikimedia.org/blob/mediawiki%2Fselenium.git/HEAD/lib%2Fmediawiki_selenium%2Fbrowser_factory%2Fbase.rb#L64 [23:07:58] marxarelli, where does it get lower-cased? [23:08:38] matt_flaschen: 1.x is a little different is how it loads environment variables [23:08:54] first it loads settings from an environment.yml, then from ENV [23:09:05] all the variable names are normalized to be lowercase symbols [23:26:39] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<11.11%) [23:31:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:31:25] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [23:31:43] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:46:40] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK