[00:25:07] 06Release-Engineering-Team, 07Browser-Tests, 03Reading-Web-Sprint-94, 07Unplanned-Sprint-Work: Invalid CSRF token in Selenium browser test in Popups/MobileFrontend blocking merges - https://phabricator.wikimedia.org/T160566#3104709 (10Jdlrobson) [00:25:18] 06Release-Engineering-Team, 07Browser-Tests, 03Reading-Web-Sprint-94, 07Unplanned-Sprint-Work: Invalid CSRF token in Selenium browser test in Popups/MobileFrontend blocking merges - https://phabricator.wikimedia.org/T160566#3103750 (10Jdlrobson) p:05Triage>03High Seen also in https://integration.wikime... [00:37:31] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [01:31:04] 06Release-Engineering-Team, 07Browser-Tests, 03Reading-Web-Sprint-94, 07Unplanned-Sprint-Work: Invalid CSRF token in Selenium browser test in Popups/MobileFrontend blocking merges - https://phabricator.wikimedia.org/T160566#3104840 (10EddieGP) p:05High>03Unbreak! Looking into the recent changes dashboa... [04:04:33] PROBLEM - Puppet staleness on deployment-aqs02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [04:18:24] Yippee, build fixed! [04:18:25] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #331: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/331/ [05:38:07] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 03Language-2017 Sprint 3, and 7 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3104982 (10KartikMistry) It looks https://integration.wikimedia.org/ci/job/mwext-qunit-jessi... [06:51:25] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #301: 04FAILURE in 2 hr 11 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/301/ [07:02:35] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:52:21] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 13Patch-For-Review: Jenkins Browser tests for Wikibase are failing - https://phabricator.wikimedia.org/T160519#3105150 (10hashar) I eventually approved the revert patch to unblock development hitting that nasty corner case. [07:55:58] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 13Patch-For-Review: Jenkins Browser tests for Wikibase are failing - https://phabricator.wikimedia.org/T160519#3105153 (10hashar) [07:56:01] 06Release-Engineering-Team, 07Browser-Tests, 03Reading-Web-Sprint-94, 07Unplanned-Sprint-Work: Invalid CSRF token in Selenium browser test in Popups/MobileFrontend blocking merges - https://phabricator.wikimedia.org/T160566#3105155 (10hashar) [07:57:19] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 13Patch-For-Review: Jenkins Browser tests for Wikibase/Popups etc are failing: Invalid CSRF token in Selenium browser - https://phabricator.wikimedia.org/T160519#3101972 (10hashar) a:03hashar [07:58:47] 06Release-Engineering-Team, 07Browser-Tests, 03Reading-Web-Sprint-94, 07Unplanned-Sprint-Work: Invalid CSRF token in Selenium browser test in Popups/MobileFrontend blocking merges - https://phabricator.wikimedia.org/T160566#3105163 (10hashar) WMDE has hit that on Wednesday with browser tests and task T1605... [09:27:07] (03PS1) 10Volans: [operations/switchdc] add tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/343005 [10:52:27] (03CR) 10Hashar: "wdio thing is only on master branch for now so should only trigger on the master branch." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/324719 (https://phabricator.wikimedia.org/T139740) (owner: 10Zfilipin) [11:03:12] yo [11:03:27] has the wikibase gate and submit job hung? [11:03:30] https://integration.wikimedia.org/zuul/ [11:06:50] nope [11:06:54] it's getting through it [11:15:58] (03PS3) 10Paladox: Update composer to 1.3.2 [integration/composer] - 10https://gerrit.wikimedia.org/r/340791 (https://phabricator.wikimedia.org/T125343) [11:16:12] (03CR) 10Paladox: "Bump" [integration/composer] - 10https://gerrit.wikimedia.org/r/340791 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [12:27:12] (03CR) 10Reedy: "1.4.1 is out. This isn't the latest release" [integration/composer] - 10https://gerrit.wikimedia.org/r/340791 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [12:38:43] Hello releng hope everyone having a good day [12:46:14] hashar: quick question do you want me to assign T158652 to you as you seem to be handling the depooling of the instances? [12:46:14] T158652: Depool precise jenkins instances - https://phabricator.wikimedia.org/T158652 [12:51:15] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Depool precise jenkins instances - https://phabricator.wikimedia.org/T158652#3105862 (10hashar) a:03hashar [12:51:18] Zppix: indeed. Idid it ! [12:51:25] hashar: ack :) [12:51:31] Zppix: will close it next monday [12:51:51] hashar: no rush, take your time, jenkins is no easy thing to do. [13:12:45] out of curiosity is there a hack that would allow me to run a maint script on group2 wikis with wmf16 code base? [13:15:26] dcausse: --wiki=enwiki is passed by mwscript to: multiversion/MWScript.php [13:15:42] which lookup the version to use based on wikiversion.json / wikiversion.php [13:15:57] so potentially you could manually hack the version in those files [13:16:13] but any process depending on those files would also be switched [13:16:39] yes so I would need to find an isolated machine with mediawiki and hack wikiversion.json [13:16:47] I guess [13:16:57] maybe mwdebug1002 could be used [13:17:09] or backport the code to wmf.15 ? :] [13:17:24] (I am kidding, I guess it is for the elastic search 5 thing) [13:17:26] hm not really possible :) [13:17:35] then [13:17:45] mwdebug1001 do not let you runs mwscript [13:17:52] oh that's right :/ [13:17:59] mwscript is hardcoded to invoke "php5" [13:18:14] and the box is an app server so it does not have all the Zend extensions needed [13:18:15] but [13:18:40] you can probably forge the proper command using "php" instead of php5 [13:18:53] (or copy mwscript and edit it [13:18:58] (03CR) 10Gehel: [C: 04-1] Search: add checkstyle:check to the job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/342878 (owner: 10DCausse) [13:19:09] ok I see [13:19:49] alternatively there is the equivalent of terbium: wasat.codfw.wmnet [13:19:55] which is probably has nothing running on it [13:20:17] yes, I rarely use this one, not sure if someone has scheduled some crons there [13:20:35] that I cant tell [13:21:04] ok, wasat would be nice in fact, it'll be way faster since I'd like to prepare some indices in codfw [13:21:22] \O/ [13:21:34] this is not strictly needed but would give us more flexibility if something wrong happens when group2 will switch to elastic5 [13:21:51] wasat should have maintenance script installed [13:21:57] yes [13:21:59] and I have no idea whether it has cron enabled [13:22:05] will have a look [13:23:11] hashar: thanks for the info, I'll talk with Erik to see if it's worth doing this hack before wmf16 rolls out to group2 [13:24:47] dcausse: and the script multiversion/updateWikiversions [13:25:02] would let one migrate a given dblist from a version to another [13:25:41] which iirc is just about updating the .json file and regenerating the wikiversions.php [13:25:42] hashar: oh I see, it'd be even cleaner, will scap pull revert my hack when I'm done? [13:26:04] hopefully yeah [13:26:08] ok nice [13:26:22] the huge gotcha is that wasat is shared [13:26:29] and might potentially have a script running on it [13:26:36] yes I'll have to double check that [13:26:43] that's the main concern [13:27:10] if I do this I'll have to carefully warn people not to use it for a couple of hours [13:27:18] or we craft a patch that lets us override the version to use for mwscript/multiversion. But really I dont know how many things are relaying on that [13:27:44] and potentially ops can update the motd to send some large bold RED text in the login motd :] [13:27:53] yes we could :) [13:27:58] it is not used much at least [13:28:11] so maybe just a mail to ops-l is sufficient [13:28:18] sounds good [13:28:32] and we switch it tonight, so that would be for a limited time [13:29:00] yes this is really to prepare an index "just in case" the load is too high [13:30:44] heading out to have a nap [13:38:20] (03PS2) 10DCausse: Search: switch verify instead of package [integration/config] - 10https://gerrit.wikimedia.org/r/342878 [13:38:39] (03CR) 10DCausse: Search: switch verify instead of package (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/342878 (owner: 10DCausse) [13:39:23] (03PS3) 10DCausse: Search: switch to verify instead of package [integration/config] - 10https://gerrit.wikimedia.org/r/342878 [13:49:09] PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:56:20] (03CR) 10Gehel: [C: 031] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/342878 (owner: 10DCausse) [13:56:23] 06Release-Engineering-Team, 10MediaWiki-Vagrant, 06Operations, 07Epic, 13Patch-For-Review: [EPIC] Migrate base image to Debian Jessie - https://phabricator.wikimedia.org/T136429#3106025 (10Gilles) [15:03:11] !log deployment-prep setting role::logging::mediawiki::udp2log::rotate: 15 in project wide hiera configuration [15:03:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:42:12] (03PS1) 10Hashar: oojs-ui-demos-publish to Jessie/modern NodeJs [integration/config] - 10https://gerrit.wikimedia.org/r/343077 (https://phabricator.wikimedia.org/T160583) [15:43:25] hashar: You're awesome, BTW. Remind me to buy you a drink in Vienna. [15:44:23] 10Continuous-Integration-Config, 10OOjs-UI, 13Patch-For-Review: oojs-ui-demos-publish job needs to run on anything other than Trusty please (modern Node version needed) - https://phabricator.wikimedia.org/T160583#3104266 (10hashar) I have updated the job based on the CI config change above. I have retrigger... [15:45:11] (03CR) 10Hashar: "I have refreshed the job. Pending outcome on oojs/ui change https://gerrit.wikimedia.org/r/343028" [integration/config] - 10https://gerrit.wikimedia.org/r/343077 (https://phabricator.wikimedia.org/T160583) (owner: 10Hashar) [15:53:17] Yippee, build fixed! [15:53:17] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #361: 09FIXED in 31 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/361/ [15:54:26] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:15:52] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), 05MW-1.29-release-notes, 13Patch-For-Review: Jenkins Browser tests for Wikibase/Popups etc are failing: Invalid CSRF token... - https://phabricator.wikimedia.org/T160519#3106701 [17:25:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10OOjs-UI: Speed up oojs/ui Jenkins jobs - https://phabricator.wikimedia.org/T155483#3107062 (10thcipriani) [17:27:49] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Create "High Priority" test pipeline - https://phabricator.wikimedia.org/T160667#3107070 (10thcipriani) [17:32:13] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Create "High Priority" gate-and-submit pipeline - https://phabricator.wikimedia.org/T160668#3107085 (10thcipriani) [17:33:09] nice, little steps sprint [17:35:03] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Remove zend tests from wmf branches - https://phabricator.wikimedia.org/T160669#3107103 (10thcipriani) [17:55:13] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Do not say "< wmf-insecte> Yippee, build fixed!" - https://phabricator.wikimedia.org/T95395#1188546 (10bd808) My $0.02 USD: WONTFIX and move on. Fix your tests so they aren't flaky, test locally before uploadin... [18:52:38] twentyafterfour: fyi i hacked up wasat.codfw.wmnet to point all wikis at wmf.16, so i could start building the CirrusSearch completion suggesters before the rollout and be able to flip them on sooner than later. If syncing out a change that rolls everything forward i don't forsee any issues, but if you need to do a rollback ping me afterwards so i can check which wikis ended up trying to build with the wmf.15 code [18:57:37] ebernhardson: ok [19:03:16] so, if this gets merged: https://gerrit.wikimedia.org/r/#/c/342913 will it be deployed automatically? [19:05:03] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet fails on integration instances: nfs_mount[home-on-labstoresvc]: umount: /home: not mounted - https://phabricator.wikimedia.org/T155820#3107512 (10chasemp) 05Open>03Resolved a:03chasemp [19:05:07] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), 05MW-1.29-release-notes, 13Patch-For-Review: Jenkins Browser tests for Wikibase/Popups etc are failing: Invalid CSRF token... - https://phabricator.wikimedia.org/T160519#3107514 [19:15:04] (03PS1) 10Awight: Run tox for new repo [integration/config] - 10https://gerrit.wikimedia.org/r/343113 [19:33:26] milimetric: no, you have to put it up for swat or wait until next week's train [19:39:02] define "automatically" :) [19:40:10] damn people [19:41:46] next week sounds fine to me [19:42:12] that's "automatically" enough for sure :) Now to chase down someone to +2 so I don't have to self-merge [19:50:53] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Labs instance ci-jessie-wikimedia-498353 can not be deleted - https://phabricator.wikimedia.org/T156636#3107666 (10chasemp) 05Open>03Resolved a:03chasemp marking resolved as this may be an artifact of bad config gone by but let's t... [20:32:22] (03PS4) 10Paladox: Update composer to 1.4.1 [integration/composer] - 10https://gerrit.wikimedia.org/r/340791 (https://phabricator.wikimedia.org/T125343) [20:32:48] 10Continuous-Integration-Infrastructure, 07Composer, 13Patch-For-Review: Upgrade integration/composer to 1.4.1 stable - https://phabricator.wikimedia.org/T125343#3107848 (10Paladox) [20:56:41] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), 05MW-1.29-release-notes, 13Patch-For-Review: Jenkins Browser tests for Wikibase/Popups etc are failing: Invalid CSRF token... - https://phabricator.wikimedia.org/T160519#3107931 [21:43:14] (03CR) 10Thcipriani: [C: 032] Run tox for new repo [integration/config] - 10https://gerrit.wikimedia.org/r/343113 (owner: 10Awight) [21:44:58] (03Merged) 10jenkins-bot: Run tox for new repo [integration/config] - 10https://gerrit.wikimedia.org/r/343113 (owner: 10Awight) [21:46:46] thcipriani: thanks! [21:46:55] awight: yw :) [21:47:04] getting ready to deploy right now [21:47:49] Great, I'll take a break from sippy-birding +2+2/submit [21:48:21] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/343113/ [21:48:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:48:51] :) [21:49:04] awight: should be live now [22:21:20] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), 05MW-1.29-release-notes, 13Patch-For-Review: Jenkins Browser tests for Wikibase/Popups etc are failing: Invalid CSRF token... - https://phabricator.wikimedia.org/T160519#3101972 [22:43:26] Anyone happen to know, if there is configuration for tox.ini which will show the contents of failed assertions like "pytest -v -v" does? [22:50:50] thcipriani: Is it still possible to ssh to the CI nodes? [22:51:08] I'm getting a failed assertion but no details, so I wanted to run the tests by hand. [22:51:08] awight: blerg not for tox jobs. [22:51:12] d'oh [23:10:05] 06Release-Engineering-Team, 10MediaWiki-Authentication-and-authorization, 05MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), 05MW-1.29-release-notes, 13Patch-For-Review: Jenkins Browser tests for Wikibase/Popups etc are failing: Invalid CSRF token... - https://phabricator.wikimedia.org/T160519#3108625