[00:00:55] chrismcmahon,yeah just noticed that [00:00:56] greg-g: right on, accepted [00:01:47] ryasmeen: seems to date from yesterday [00:02:57] * chrismcmahon splits attention between writing tests and reading tests, and sometimes one or the other falls down for a bit [00:07:09] chrismcmahon: yeah, Roan is going to look into that, and it happens in iOS safari too [00:12:07] ryasmeen: maybe we will put your email on the list for when VE and MF browser test builds fail :-) [00:12:24] since they're looking pretty good right now [00:13:18] Sure chrismcmahon, that will be great. [00:34:36] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #173: FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/173/ [00:52:32] heya, rubocop failing for Flow with "02:52:14 Gemfile:1:1: C: Missing space after #. [00:52:35] 02:52:14 #ruby=ruby-2.1.1" [00:52:55] but all other Gemfiles have no space after # on their lines. [00:59:13] https://github.com/wikimedia/mediawiki-extensions-CirrusSearch/blob/master/.rubocop.yml solves it: [00:59:16] Style/LeadingCommentSpace: [00:59:19] Exclude: [00:59:21] - Gemfile # RVM doesn't recognise spaces after the #s [02:00:42] (03CR) 10Jforrester: [C: 031] mwconf: Remove references to pmtpa [integration/jenkins] - 10https://gerrit.wikimedia.org/r/173518 (owner: 10Krinkle) [03:07:27] (03PS1) 10Krinkle: Add mwext-Flow-npm job [integration/config] - 10https://gerrit.wikimedia.org/r/174359 [03:09:27] (03PS3) 10Krinkle: Migrate mediawiki/qunit jobs from production slaves to labs [integration/config] - 10https://gerrit.wikimedia.org/r/173529 (https://bugzilla.wikimedia.org/72063) [03:12:04] (03PS4) 10Krinkle: Migrate mediawiki/qunit jobs from production slaves to labs [integration/config] - 10https://gerrit.wikimedia.org/r/173529 (https://bugzilla.wikimedia.org/72063) [03:12:21] (03CR) 10Krinkle: Migrate mediawiki/qunit jobs from production slaves to labs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/173529 (https://bugzilla.wikimedia.org/72063) (owner: 10Krinkle) [03:13:36] (03CR) 10Jforrester: [C: 031] "LGTM." [integration/config] - 10https://gerrit.wikimedia.org/r/173529 (https://bugzilla.wikimedia.org/72063) (owner: 10Krinkle) [03:19:17] Great. jenkins-job-builder is no longer compatible with the default python that ships with Mac OS [03:19:18] https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#What_could_go_wrong [03:28:46] (03CR) 10Krinkle: "Affected jobs that need recompiling/deploying:" [integration/config] - 10https://gerrit.wikimedia.org/r/173529 (https://bugzilla.wikimedia.org/72063) (owner: 10Krinkle) [03:32:56] Krinkle: Fun! [03:34:59] PROBLEM - CI: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: integration.integration-puppetmaster.diskspace._var.byte_avail.value (20.00%) [04:36:39] RECOVERY - CI: Low disk space on /var on labmon1001 is OK: OK: All targets OK [09:18:50] Yippee, build fixed! [09:18:50] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #174: FIXED in 18 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/174/ [09:42:18] PROBLEM - CI: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: integration.integration-puppetmaster.diskspace._var.byte_avail.value (11.11%) [10:11:20] 3Wikimedia / 3Quality Assurance: Quality Assurance/Browser testing/Setup instructions is out of date - 10https://bugzilla.wikimedia.org/72732#c4 (10Quim Gil) Could you specify what needs updating, Chris McMahon? Chris Koerner seemed to be willing to give a hand. [10:50:06] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #283: FAILURE in 1 hr 3 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/283/ [12:04:30] Yippee, build fixed! [12:04:31] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #58: FIXED in 27 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/58/ [12:49:33] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1009: FAILURE in 3 min 32 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1009/ [12:49:58] PROBLEM - BetaLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki01.diskspace._var.byte_avail.value (33.33%) [13:41:32] (03PS1) 10Hashar: Job template to run composer scripts [integration/config] - 10https://gerrit.wikimedia.org/r/174410 [13:41:34] (03PS1) 10Hashar: Add cdb-composer-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/174411 [13:41:36] (03PS1) 10Hashar: Add customization of publishers for composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/174412 [13:58:14] (03PS2) 10Hashar: Replace cdb-phpunit with cdb-composer-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/174411 [13:58:36] (03CR) 10Hashar: "Added Zuul configuration. The FIXME is addressed in follow up commit https://gerrit.wikimedia.org/r/#/c/174412/" [integration/config] - 10https://gerrit.wikimedia.org/r/174411 (owner: 10Hashar) [14:07:39] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #59: FAILURE in 1 min 37 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/59/ [14:26:32] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #59: ABORTED in 9 min 31 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/59/ [14:26:41] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #45: ABORTED in 4 min 40 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/45/ [14:33:04] anyone an idea what happend to http://en.wikipedia.beta.wmflabs.org/ and http://wikidata.beta.wmflabs.org? [14:37:54] 3Wikimedia / 3Quality Assurance: Write tests for mediawiki_selenium Ruby gem - 10https://bugzilla.wikimedia.org/63474#c1 (10Željko Filipin) Dan, you are working on this, right? [15:01:04] (03PS1) 10Hashar: Add job template for doxygen publishing [integration/config] - 10https://gerrit.wikimedia.org/r/174416 [15:01:06] (03PS1) 10Hashar: Publish doxygen doc for the 'cdb' project [integration/config] - 10https://gerrit.wikimedia.org/r/174417 (https://bugzilla.wikimedia.org/73530) [15:01:21] hmm, betalabs is down [15:01:24] shinken sent alerts [15:01:25] yay [15:01:39] RECOVERY - BetaLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [15:06:07] Yippee, build fixed! [15:06:07] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #60: FIXED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/60/ [15:31:37] 3Wikimedia / 3Continuous integration: Publish cdb documentation on doc.wikimedia.org - 10https://bugzilla.wikimedia.org/73530#c6 (10Antoine "hashar" Musso (WMF)) a:3Antoine "hashar" Musso (WMF) The patches above would let us generate doxygen documentation after a merge / ref update and have it published un... [15:34:10] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #60: SUCCESS in 27 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/60/ [15:40:29] Yippee, build fixed! [15:40:30] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #1010: FIXED in 28 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/1010/ [16:06:51] 3Wikimedia / 3Continuous integration: Jenkins: report metrics to statsd - 10https://bugzilla.wikimedia.org/55412#c4 (10Antoine "hashar" Musso (WMF)) We are no more using this plugin. It was meant to generate graph for the executor queues but the Jenkins plugin does not collect them. Since reporting a metric... [16:07:23] 3Wikimedia / 3Continuous integration: [upstream] Jenkins: Statsd metrics miss queue/nodes/executors informations - 10https://bugzilla.wikimedia.org/55988#c1 (10Antoine "hashar" Musso (WMF)) 5NEW>3RESO/WON We are no more using this plugin. It was meant to generate graph for the executor queues but the Jen... [16:21:05] YuviPanda: so, beta was down due to? [16:21:08] space issues? [16:21:14] no idea. [16:21:28] I woke up to emails and pings about Geshi, but.. beta's back up and that bug's still open https://bugzilla.wikimedia.org/show_bug.cgi?id=73567 [16:21:29] I saw other comments from other folks and had just woken up so let it be [16:21:34] * greg-g nods [16:21:36] it also had a space alert [16:21:38] that was resolved [16:21:40] so might be space [16:21:42] yeah [16:21:49] stupid space issue [16:21:58] seriously, that one annoys the ef out of me [16:22:05] DISK IS CHEAP [16:22:48] greg-g: indeed. [16:22:55] greg-g: give everyone a 10G /var from start, end of story [16:23:05] greg-g: others point out that 'oh no, it is a log growth problem' [16:23:09] 2G is waaaaay tooooo smalllll [16:23:15] in fact, having /var be smaller than RAM is stupid. [16:23:21] but oh well [16:23:27] new instances have resizable /var [16:23:32] so way to fix is to re-create instances [16:23:37] OpenStack limitation, etc [16:24:25] yeah, and recreating sounds so fun doesn't it? :) [16:26:02] greg-g: it does :) at least we'll fix our puppet manifests. [16:26:31] greg-g: and if we do it as 'recreate / test / decom old one' then there won't be downtime either [16:26:39] yeah [16:26:42] greg-g: we could just wait for trusty roll out and do both in one go [16:26:54] yeah, I wouldn't rush a recreation of beta [16:26:57] yeah [16:27:07] at some point prod will be all trusty, and we'll have to recreate anyway. [16:27:11] * greg-g nods [16:27:16] until then, someone has to keep doing 'rm -rf' in places, I guess :) [16:27:16] and then Jessie! :) [16:27:20] hehe [16:27:27] greg-g: do you know there's a lucid instance in beta atm? :) [16:27:47] the one from ariel? [16:28:00] I replied about that on that thread :) ask them and delete it ;) [16:28:06] or ask them *to* delete it [16:28:07] oh [16:28:10] I didn't see the reply [16:28:12] must've gotten lost [16:28:15] I did just last night [16:28:18] aaah [16:28:28] I asked, it has to be around as long as we have lucid in prod [16:28:30] which we still do [16:28:35] so that's going to be red for a while [16:28:50] oh, it's used? [16:28:52] I thought it wasn't used [16:29:06] sodium is lucid [16:29:13] and nickel was, was decomm'd last week [16:29:37] I mean in beta [16:29:45] oh [16:29:47] it's used [16:29:50] to test salt upgrades [16:29:51] :( [16:30:01] can't run puppet, and you can't login without a labs root key [16:30:02] ok, as long as it's used and known [16:30:20] so I shouldn't care about it right now, is the summary, right? :) [16:30:27] yeah, pretty much :) [16:30:29] sweet [16:30:34] so it'll be red (or 'unknown') but you can ignore that :) [16:30:46] the *other* ones you should, however :) http://shinken.wmflabs.org/problems [16:30:47] cool, gotcha, I thought red/unknown meant something worse [16:30:50] yes yes... [16:30:52] it's all betalabs now [16:30:52] :) [16:33:00] greg-g: :) re: https://phabricator.wikimedia.org/T1334 I added a patch, do poke people for some +1 before I merge. [16:33:03] I think it's a good start. [16:33:23] also icinga monitoring will go away soon, for labs. Shinken needs slightly more reliable email delivery and IRC notifs (I'm on that now), and we're all good [16:35:06] 3Wikimedia / 3Continuous integration: [OPS] Jenkins: puppet master fills /var on labs with yaml reports - 10https://bugzilla.wikimedia.org/73472#c11 (10Greg Grossmeier) Just keeping the heat on this bug, we had an outage this morning (times in Eastern US): 07:49 < icinga-wm> PROBLEM - BetaLabs: Low disk spac... [16:35:59] YuviPanda: ah, cool, and sorry for doing busy work with old technology then [16:36:18] greg-g: nah, it was a trivial patch. and we'll have it for a while anyway, until shinken proves reliable enough [16:36:19] RECOVERY - CI: Low disk space on /var on labmon1001 is OK: OK: All targets OK [16:36:22] (it's already more reliable than icinga tho) [16:36:36] cool [17:02:08] greg-g: just noting that shinken is *all* red now, and caused by OpenStack upgrade in progress. andrewbogott is monitoring. Just a FYI [17:05:52] "great" :) [17:13:23] (03CR) 10BryanDavis: "The use of `composer install` will by default install all packages in require-dev. For many projects this will include phpunit of some var" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [17:43:39] 3Wikimedia / 3Continuous integration: Zuul: scale merge operations (tracking) - 10https://bugzilla.wikimedia.org/68480 (10Andre Klapper) [17:47:07] (03CR) 10Krinkle: [C: 04-1] Job template to run composer scripts (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [17:48:52] 3Wikimedia Labs / 3deployment-prep (beta): File upload area resorts to 0777 permissions to for uploaded conent - 10https://bugzilla.wikimedia.org/73206 (10Greg Grossmeier) p:5Unprio>3High [17:51:48] (03CR) 10Krinkle: Job template to run composer scripts (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [17:53:17] OK, I have gotten rid of all the sleeps in the VE tests and also arrange the page object in alphabetical order. In hindsight, I wonder if we want to have the language_screenshot test have its own separate page object. (Not that I would do that today) [17:53:39] (03CR) 10Krinkle: "Perhaps include composer validate in here as well for projects using composer-test? Seems useful to reduce the number of jobs where conven" [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [18:25:58] Yippee, build fixed! [18:25:59] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #46: FIXED in 2 hr 44 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/46/ [18:34:42] (03CR) 10Cscott: "ping!" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/174047 (owner: 10Cscott) [19:44:35] 3Wikimedia / 3Continuous integration: [OPS] Jenkins: puppet master fills /var on labs with yaml reports - 10https://bugzilla.wikimedia.org/73472#c12 (10Antoine "hashar" Musso (WMF)) I know about two reasons for the HHVM application servers on beta cluster fill /var/ : Bug 73262 - hhvm apache fills /var/log/... [19:56:56] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #365: FAILURE in 1 hr 1 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/365/ [20:45:39] marxarelli: can you +2 this? Jenkins is messing up https://gerrit.wikimedia.org/r/#/c/174446/ [20:48:52] 3Wikimedia / 3Continuous integration: Jenkins: Add check ensuring all mediawiki-core php files have proper license headers - 10https://bugzilla.wikimedia.org/42472#c3 (10Krinkle) 5NEW>3RESO/WON p:5Low>3Lowest I'd say catch it through code review and maybe scan occasionally. I don't think it's worth t... [20:52:04] Jenkins is annoying [20:53:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #125: FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/125/ [21:05:13] 3Wikimedia / 3Continuous integration: Jenkins: Create phpcs sniff to detect text before the first 3Wikimedia / 3Continuous integration: Jenkins: Configure jobs to be runnable on any slaves - 10https://bugzilla.wikimedia.org/53594 (10Krinkle) [21:11:50] 3Wikimedia / 3Continuous integration: Jenkins: Run jobs in disposable VMs - 10https://bugzilla.wikimedia.org/45499#c15 (10Krinkle) p:5Normal>3High s:5enhanc>3normal There have been discussions about this on engineering and ops mailing list as well as some write ups and comparison on Google Docs. Wha... [21:27:40] !log Ran `GIT_SSH=/var/lib/git/ssh git pull --rebase` in deployment-salt:/srv/var-lib/git/labs/private [21:27:44] Logged the message, Master [21:50:45] 3Wikimedia / 3Continuous integration: Jenkins: Set up job to detect bad patterns (tracking) - 10https://bugzilla.wikimedia.org/47054 (10Krinkle) [21:50:45] 3Wikimedia / 3Continuous integration: Jenkins: Fail on BOM in submitted files - 10https://bugzilla.wikimedia.org/38233#c3 (10Krinkle) I don't think we should have a generic job that scans all files everywhere for a BOM. Instead let's defer this to individual linters or coding style checkers. phpcs has a sni... [21:50:45] 3Wikimedia / 3Continuous integration: Jenkins: Create phpcs sniff to detect text before the first 3Wikimedia / 3Continuous integration: Jenkins: Set up job to detect bad patterns (tracking) - 10https://bugzilla.wikimedia.org/47054 (10Krinkle) [21:51:37] 3Wikimedia / 3Continuous integration: Jenkins: Job in gate-and-submit to avoid submitting "DRAFT" or "WIP" commits - 10https://bugzilla.wikimedia.org/46860 (10Krinkle) [21:51:41] 3Wikimedia / 3Continuous integration: Jenkins: Set up job to detect bad patterns - 10https://bugzilla.wikimedia.org/47054#c1 (10Krinkle) 5NEW>3RESO/WOR This tracking bug has become obsolete as all its tasks have been or will be resolved in more integrated ways. [21:52:35] 3Wikimedia / 3Continuous integration: Jenkins: Set up jobs for SwiftCloudFiles extension - 10https://bugzilla.wikimedia.org/46262 (10Krinkle) [21:52:50] 3Wikimedia / 3Continuous integration: Jenkins: Set up job in gate-and-submit to avoid submitting "DRAFT" or "WIP" commits - 10https://bugzilla.wikimedia.org/46860 (10Krinkle) [21:53:05] 3Wikimedia / 3Continuous integration: Jenkins: Create phpcs sniff to ensure test functions have the proper signature - 10https://bugzilla.wikimedia.org/46483 (10Krinkle) [21:56:06] 3Wikimedia / 3Continuous integration: Status update in Gerrit of current queued commit/testing request - 10https://bugzilla.wikimedia.org/47527#c6 (10Krinkle) 5NEW>3RESO/WON Also, since the creation of this ticket, we've built the visual dashboard: https://integration.wikimedia.org/zuul/ Which allows o... [22:30:25] bd808: you don't use the lucid salt vm in beta right? [22:33:39] greg-g: Nope that was all Ariel [22:34:03] * bd808 pointed yuvipandajs at him previously [22:34:41] Yeah I talked to apergos [22:34:53] It is gonna stay until prod has no lucid [22:42:07] 3Wikimedia / 3Continuous integration: Jenkins: Set up job in gate-and-submit to avoid submitting "DRAFT" or "WIP" commits - 10https://bugzilla.wikimedia.org/46860#c5 (10C. Scott Ananian) (In reply to Antoine "hashar" Musso (WMF) from comment #4) > Revisiting after a while: Zuul does not have support to preve... [22:52:56] bd808: just making sure you didn't also do anythign with it [22:52:57] ty [23:20:17] cscott: While this should be easy to resolve, I suspect the reason the commit message isn't in HEAD is due to the tests running on a merge commit (by design, we want to test how the repo'd be after merge). so HEAD isn't the proposed commit [23:20:43] cscott: But I think there's a zuul ENV var with the hash, and that hash will be in the local git repo so we can do git show $ZUUL_whatev instead [23:21:08] but yeah, overal that should be trivial to do in a job. Not sure what hashar is seeing, I'll ask. Could be that we're missing something [23:54:35] 3Wikimedia / 3Quality Assurance: Write tests for mediawiki_selenium Ruby gem - 10https://bugzilla.wikimedia.org/63474#c2 (10Dan Duvall) In the env-abstraction-layer branch I've been implementing unit tests as I go,[1] but we could certainly use a handful of functional tests. [1] https://github.com/wikimedia... [23:56:49] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #300: FAILURE in 1 hr 23 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/300/