[04:05:19] Yippee, build fixed! [04:05:19] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #56: 09FIXED in 9 min 18 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/56/ [10:36:39] 07Browser-Tests, 10VisualEditor, 13Patch-For-Review, 15User-zeljkofilipin: Fix language screenshots browser test on VisualEditor - https://phabricator.wikimedia.org/T119482#2408948 (10Esanders) [10:39:22] 07Browser-Tests, 10VisualEditor, 13Patch-For-Review, 15User-zeljkofilipin: Fix language screenshots browser test on VisualEditor - https://phabricator.wikimedia.org/T119482#2408955 (10Esanders) [11:21:02] (03PS1) 10Addshore: DNM Add extension-selenium to RevisionSlider [integration/config] - 10https://gerrit.wikimedia.org/r/296208 (https://phabricator.wikimedia.org/T133278) [12:25:52] Project beta-scap-eqiad build #108672: 04FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/108672/ [12:34:57] Project beta-scap-eqiad build #108673: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/108673/ [12:44:58] Project beta-scap-eqiad build #108674: 04STILL FAILING in 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/108674/ [12:56:02] Yippee, build fixed! [12:56:03] Project beta-scap-eqiad build #108675: 09FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/108675/ [13:09:30] hi, I made a mistake when releasing a java artifact and I'd like to delete a tag from gerrit, but gerrit rejects the command saying: "You need 'Push' rights with the 'Force Push'" [13:33:45] dcausse: I'm tempted to say: don't delete the tag, just create a new one. [13:33:59] (that's the policy I had previously) [13:47:40] Yippee, build fixed! [13:47:40] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #63: 09FIXED in 3 min 39 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/63/ [13:53:35] gehel: yes I created a new "-fixed" one... [13:53:59] I would have incremented version number... but that works as well [15:01:35] greg-g Hi [15:09:22] paladox: if you need something, please just say it [15:09:35] hi, btw [15:09:47] hi :) [15:09:53] greg-g: Sorry, yes, could you have someone test https://gerrit.wikimedia.org/r/#/c/296138/ please [15:10:20] Since yesturday mutante merged the patch that install's the rewrites onto phabricator but it broke bugzilla [15:10:34] we will get to it [15:10:50] Ok, thanks. [15:10:50] we know, we also read phabricator notifications [15:10:54] ok [15:11:13] thanks [15:38:17] Yippee, build fixed! [15:38:18] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #59: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/59/ [15:45:03] Yippee, build fixed! [15:45:04] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #59: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/59/ [15:51:22] (03CR) 10EBernhardson: "needs rebase" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/293966 (owner: 10Lethexie) [15:57:25] (03CR) 10EBernhardson: "The phpdoc inside functions is found within core, and the usage looks sane to me, i think we probably should allow it." (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295895 (owner: 10Lethexie) [15:58:28] (03CR) 10EBernhardson: "some trivial indentation issues, but overall looks great" (033 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295115 (owner: 10Lethexie) [16:17:59] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T136973#2409503 (10mmodell) a:05thcipriani>03mmodell [17:01:49] http://www.libdems.org.uk/ [17:01:55] Sorry wrong place [17:26:57] bd808 hi, could you review https://gerrit.wikimedia.org/r/296250 [17:26:57] please [17:27:21] It would stop cases by people who add the file and then break the ci. [17:28:01] but it would also break anyone using vendor with a newer version of PHP if an autoload_static.php was created [17:28:15] didn't we already abandon this same patch? [17:29:01] bd808 hi, no it woulden [17:29:06] since it generates them both [17:29:23] Actually yes your right [17:29:25] sorry [17:29:30] but if it is not committed then the file will be missing for PHP 5.6+ users [17:29:43] I just realised that, sorry about that. [17:29:57] greg-g: Was a decision made as to the fate of wmf.7 in today's meeting? [17:30:41] paladox: the fix for this is to fix the actual linting jobs run by CI and also scap. Anything other than that is a hack that will not work [17:30:56] bd808 ij [17:31:01] ok [17:37:07] RoanKattouw: we haven't seen much movement on the blocking tasks, twentyafterfour will be/has pinged ori about it [17:37:48] I ask because (and I probably should have told you about this, oops) we are planning a big Echo change for wmf.9, so we want to merge those patches as soon as wmf.8 is cut [17:38:31] gotcha [17:40:05] I'm writing up a more detailed deployment plan for this (to be executed by Matt and Stephane, as I will be on vacation next week), so the more certainty there is about branch cut and deployment schedules the better [17:40:41] We don't rely on MW core changes so we can always sidestep the train using submodule updates if needed, but it would be nice to know what's gonna happen train-wise [17:40:47] (Mukunda has pinged on the relevant task) [17:41:02] OK [17:41:14] I guess the train cannot resume while the issue persists [17:41:56] Does it seem likely that there will be no train this week, and that wmf.8 will be postponed a week (or more) until the issue is resolved? [17:43:37] without input from ori or matma I can't say, but am leaning towards postponed (since that's the default) [17:53:41] (03CR) 10EBernhardson: [C: 032] Comments should start with new line. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295901 (owner: 10Lethexie) [17:54:32] (03Merged) 10jenkins-bot: Comments should start with new line. [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295901 (owner: 10Lethexie) [18:09:24] !log Git repo at deployment-puppetmaster.deployment-prep:/var/lib/git/labs/private is behind upstream due to multiple modules/secret/secrets/keyholder local files that would be overwritten by upstream changes. [18:09:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:09:52] somebody who knows what those ^ are for should probably turn them into local commits [18:15:08] * thcipriani looks at labs/private [18:20:07] !log deployment-puppetmaster.deployment-prep:/var/lib/git/labs/private modules/secret/secrets/keyholder keys conflicts resolved [18:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:21:16] 10Continuous-Integration-Config, 07I18n, 07Tracking: Configure banana checker for i18n files to run on all MediaWiki extensions and skins (tracking) - https://phabricator.wikimedia.org/T94547#2409815 (10Seb35) [18:48:57] 06Release-Engineering-Team, 06Multimedia, 10Wikimedia-General-or-Unknown: Spike of "unknown" errors experienced by users of UploadWizard after wmf.7 deployment - https://phabricator.wikimedia.org/T138585#2404518 (10greg) @Anomie @matmarex should this block further wmf.7 rollout? [18:55:17] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Upgrade mariadb in deployment-prep and use GTID - https://phabricator.wikimedia.org/T138778#2409864 (10aaron) [19:11:17] 06Release-Engineering-Team, 06Multimedia, 10Wikimedia-General-or-Unknown: Spike of "unknown" errors experienced by users of UploadWizard after wmf.7 deployment - https://phabricator.wikimedia.org/T138585#2409896 (10Anomie) Ideally we'd add logging and redeploy it at least long enough to get backtraces. [19:19:40] 06Release-Engineering-Team, 06Multimedia, 10Wikimedia-General-or-Unknown: Spike of "unknown" errors experienced by users of UploadWizard after wmf.7 deployment - https://phabricator.wikimedia.org/T138585#2409930 (10greg) >>! In T138585#2409896, @Anomie wrote: > Ideally we'd add logging and redeploy it at lea... [19:24:42] 06Release-Engineering-Team, 03releng-201617-q1, 15User-greg: Perform a technical debt analysis of software and services maintained by WMF Release Engineering - https://phabricator.wikimedia.org/T138225#2409942 (10greg) [19:24:44] 06Release-Engineering-Team: Review and prioritize technical debt owed by RelEng (summer 2016) - https://phabricator.wikimedia.org/T137840#2409941 (10greg) [19:25:36] 10MediaWiki-Releasing, 03releng-201516-q4, 05MW-1.27-release, 05Release: [keyresult] Release MW 1.27 - https://phabricator.wikimedia.org/T129303#2409944 (10greg) [19:25:43] hi hashar :) [19:27:47] bd808: the labs/private pending stuff on beta are probably related to services switching to scap3 (poke thcipriani and twentyafterfour ) [19:27:57] we had a bunch of issues with ssh keys while rolling scap3 on beta [19:28:12] so I guess the keys got added by multiple people and left behind uncommited [19:28:17] looks like there is a commit now :) [19:28:24] hashar: *nod* I think thcipriani fixed it up after I whined [19:28:26] hashar: yarp. that's what it turned out to be. I added a commit for them :) [19:28:33] \O/ [19:28:38] bd808: thank you for the catch, appreciated. [19:28:41] bd808: thank you for whining !!! :) [19:28:49] * bd808 makes the world better by being passive agressive [19:29:05] :D [19:29:08] most probably none of us would have ever noticed [19:29:34] I just saw it because I was running the puppet master sync script manually [19:30:14] on other news, what do we do with 1.28.0-wmf.7 being blocked on the edit save time regression ? [19:30:36] I wanted to clear that up earlier today but got sink into huge amount of unrelated paper work today :( [19:30:39] 06Release-Engineering-Team, 06Team-Practices: One year review of RelEng offsite outcomes (April 2016) - https://phabricator.wikimedia.org/T112763#2409965 (10greg) @KLans_WMF I'm assuming re-doing what you did (interviewing/surveying customers of RelEng to get all that feedback before our team offsite) isn't ea... [19:31:06] hashar: I think greg-g was poking ori to see how the investigation was going [19:31:53] I saw a patch come through for it. I think the latest is on the deployment blockers ticket. [19:33:04] """Move the filter pre-caching outside of the DB lock""" https://gerrit.wikimedia.org/r/296255 [19:33:07] yeah, aaron had a patch (merged by ori), I asked aaron if he's comfortable rolling forward with it [19:33:07] that looks magic :) [19:33:33] it is too late for me to reasonably push it [19:33:41] ah, "sure" https://phabricator.wikimedia.org/T138550#2409971 [19:33:42] but I definitely back up the idea of pushing today :) [19:34:11] twentyafterfour: ^ aaron says to roll forward with wmf.7 with his new merged patch in AbuseFilter [19:37:18] but, neither anomie nor matma have moved forward with the uploadwizard/api task :/ [19:37:47] hmm [19:40:31] I posted a few links to Graphite/Grafana that can help monitor stashedit madness https://phabricator.wikimedia.org/T138550#2410000 [19:41:13] !log Rebooting deployment-logstash3.eqiad.wmflabs via wikitech. Console log full of blocked kworker messages, ssh non-responsive, and blocking logstash records being recorded. [19:41:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:45:32] 06Release-Engineering-Team, 06Multimedia, 10UploadWizard, 10Wikimedia-General-or-Unknown: Spike of "unknown" errors experienced by users of UploadWizard after wmf.7 deployment - https://phabricator.wikimedia.org/T138585#2410005 (10greg) p:05Triage>03Unbreak! For the record, this now blocks further roll... [19:48:56] 06Release-Engineering-Team: (To discuss) Implement "Request time to response" KPI - https://phabricator.wikimedia.org/T108993#2410018 (10greg) 05Open>03declined declining for now [19:53:54] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog: Don't send alerts when beta cluster is in known states that make tests fail - https://phabricator.wikimedia.org/T136172#2410022 (10greg) [19:57:12] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T137492#2369688 (10hashar) @mmodell Do you want me to cut the branch / apply the patches tomorrow during my early afternoon? Let me know by replying there and I will be happy to... [20:00:08] 06Release-Engineering-Team, 10DBA, 06Performance-Team, 07Availability, and 2 others: MASTER_POS_WAIT() alternative that works cross-DC - https://phabricator.wikimedia.org/T135027#2410031 (10ori) @jcrespo, will you be adding this to the beta cluster as well? [20:14:13] 06Release-Engineering-Team, 06Multimedia, 10UploadWizard, 10Wikimedia-General-or-Unknown: Spike of "unknown" errors experienced by users of UploadWizard after wmf.7 deployment - https://phabricator.wikimedia.org/T138585#2410069 (10matmarex) For the record, this is not a UBN for #multimedia, and I don't con... [21:07:01] 06Release-Engineering-Team, 06Multimedia, 10UploadWizard, 10Wikimedia-General-or-Unknown: Spike of "unknown" errors experienced by users of UploadWizard after wmf.7 deployment - https://phabricator.wikimedia.org/T138585#2410263 (10greg) p:05Unbreak!>03High >>! In T138585#2410069, @matmarex wrote: > For... [21:09:48] 07Browser-Tests, 06Reading-Web-Backlog, 03Reading-Web-Sprint-75-Quantitative-Paralysis: [Spike] [2 hours] Run browser tests on beta cluster from desktop domain - https://phabricator.wikimedia.org/T130429#2410270 (10Jdlrobson) @bmansurov seems actionable as is (i.e. not a spike)? What are you not sure about?... [21:11:04] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 15User-zeljkofilipin: Firefox v47 breaks mediawiki_selenium - https://phabricator.wikimedia.org/T137561#2410274 (10greg) [21:11:06] 07Browser-Tests, 06Release-Engineering-Team, 10MobileFrontend, 06Reading-Web-Backlog: Cucumber tests won't run locally on firefox 47 - https://phabricator.wikimedia.org/T138095#2410273 (10greg) [21:11:35] 07Browser-Tests, 10Browser-Tests-Infrastructure, 10MobileFrontend, 06Reading-Web-Backlog: Cucumber tests won't run locally on firefox 47 - https://phabricator.wikimedia.org/T138095#2389003 (10greg) [21:12:21] 10Deployment-Systems, 10MediaWiki-Releasing: make-wmf-branch instance requires ssh auth forwarding - https://phabricator.wikimedia.org/T126335#2410289 (10greg) [21:13:10] 06Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10VisualEditor: i18n string not updated in the Hebrew Wikipedia, even though it was translated - https://phabricator.wikimedia.org/T134072#2253949 (10greg) >>! In T134072#2262741, @Nikerabbit wrote: > I don't quite have time to investigat... [21:14:07] 10Deployment-Systems, 03Scap3, 06Operations: setup automatic deletion of old l10nupdate - https://phabricator.wikimedia.org/T130317#2410301 (10greg) [21:15:00] 10Beta-Cluster-Infrastructure: Move `tmpdir` on deployment-db instances - https://phabricator.wikimedia.org/T117446#2410307 (10greg) [21:16:14] 06Release-Engineering-Team: Reduce production log errors to zero* - https://phabricator.wikimedia.org/T115630#2410324 (10greg) [21:16:32] 06Release-Engineering-Team: [EPIC] Reduce production log errors to zero* - https://phabricator.wikimedia.org/T115630#1728605 (10greg) [21:33:36] Project selenium-Wikidata » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #37: 04FAILURE in 2 hr 43 min: https://integration.wikimedia.org/ci/job/selenium-Wikidata/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/37/ [21:55:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:03:31] 10Deployment-Systems, 03Scap3, 07WorkType-NewFunctionality: Scap3 submodule space issues - https://phabricator.wikimedia.org/T137124#2410494 (10thcipriani) [22:32:05] 06Release-Engineering-Team (Deployment-Blockers), 13Patch-For-Review, 05Release: MW-1.28.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T136973#2410576 (10greg) [22:32:52] !log deployment-prep deployed gerrit.wikimedia.org/r/296279 to puppetmaster to test kibana4 role [22:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master