[00:10:43] 7Browser-Tests, 10MediaWiki-Authentication-and-authorization: Create some end-to-end tests for SessionManager - https://phabricator.wikimedia.org/T125599#1992417 (10bd808) @greg I completely agree that this ended up in the planning email but both @anomie and I are a bit at a loss as to what @dduvall was sugges... [00:12:10] 7Browser-Tests, 10MediaWiki-Authentication-and-authorization: Create some end-to-end tests for SessionManager - https://phabricator.wikimedia.org/T125599#1992420 (10greg) Yep, that's the first step: clarifying purpose and plan. [00:15:42] bd808: I'm not pushing a specific outcome, just getting things documented; determining the "what" should be done by the smart people in the group [00:15:55] and the how and whatnot [00:16:13] sure. anomie was just surprised to have it assigned to him :) [00:16:25] understood :) [00:16:42] I just put him since ya'll are on point, and I picked a rando person among ya'll [00:16:52] oh, no, it was because that's what was said in the etherpad/email [00:17:34] whatever, smart people can come up with the real plan and do it [00:17:42] kk ;) [00:23:05] This test failed against VE on 1.27.0-wmf.12 [00:23:05] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/28844/consoleFull [00:23:34] failures are all from wikibase [00:26:07] I wonder if https://gerrit.wikimedia.org/r/#/c/267892/ is relevant [00:37:56] 7Browser-Tests, 10MediaWiki-Authentication-and-authorization: Create some end-to-end tests for SessionManager - https://phabricator.wikimedia.org/T125599#1992520 (10bd808) [01:04:27] (03PS1) 10Legoktm: Use a wrapper parameter-function so jobs can have multiple functions [integration/config] - 10https://gerrit.wikimedia.org/r/268031 (https://phabricator.wikimedia.org/T125498) [01:16:26] 7Browser-Tests, 10MediaWiki-Authentication-and-authorization: Create some end-to-end tests for SessionManager - https://phabricator.wikimedia.org/T125599#1992624 (10dduvall) It was sort of a stab in the dark, but my thought was to try performing a high number of concurrent logins on testwiki or test2wiki—or so... [01:30:36] 10Beta-Cluster-Infrastructure, 10Staging, 6Collaboration-Team-Backlog, 10DBA: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#1992641 (10Mattflaschen) [01:30:38] 10Beta-Cluster-Infrastructure, 10Flow, 3Collaboration-Team-Current: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#1992640 (10Mattflaschen) [01:31:06] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 3Collaboration-Team-Current: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#1202118 (10Mattflaschen) [03:15:28] Project beta-scap-eqiad build #88441: 04FAILURE in 0.84 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/88441/ [03:31:32] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:32:07] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:32:59] PROBLEM - Puppet failure on deployment-ms-be01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:33:38] PROBLEM - Puppet failure on deployment-ms-be02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:34:04] PROBLEM - Free space - all mounts on deployment-fluorine is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine.diskspace._srv.byte_percentfree (<100.00%) [03:34:45] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<100.00%) [03:35:07] PROBLEM - Puppet failure on deployment-cache-parsoid05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:35:13] PROBLEM - Free space - all mounts on integration-slave-trusty-1011 is CRITICAL: CRITICAL: integration.integration-slave-trusty-1011.diskspace._mnt.byte_percentfree (<100.00%) [03:35:13] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:35:17] PROBLEM - Puppet failure on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:35:57] PROBLEM - Puppet failure on cache-rsync is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:35:58] PROBLEM - Puppet failure on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:36:23] PROBLEM - Puppet failure on deployment-cache-mobile04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [03:42:23] Yippee, build fixed! [03:42:23] !log Ran deployment-bastion.deployment-prep:/home/bd808/cleanup-var-crap.sh and freed 565M [03:42:23] Project beta-scap-eqiad build #88443: 09FIXED in 7 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/88443/ [03:42:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:44:09] !log Freed 28G by deleting deployment-fluorine:/srv/mw-log/archive/*2015* [03:44:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:45:50] !log Puppet failing on deployment-fluorine with "Error: Could not set uid on user[datasets]: Execution of '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists" [03:45:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:53:58] RECOVERY - Free space - all mounts on deployment-fluorine is OK: OK: All targets OK [03:54:52] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [05:04:15] (03CR) 10Legoktm: [C: 04-2] "I'm going to re-do this piecemeal." [integration/config] - 10https://gerrit.wikimedia.org/r/267911 (owner: 10Paladox) [05:05:49] (03PS1) 10Legoktm: Set up php55lint jobs, adapt phplint macro for $PHP_BIN (re-do) [integration/config] - 10https://gerrit.wikimedia.org/r/268047 [05:10:40] (03PS2) 10Legoktm: Set up php55lint jobs, adapt phplint macro for $PHP_BIN (re-do) [integration/config] - 10https://gerrit.wikimedia.org/r/268047 [05:13:08] (03CR) 10jenkins-bot: [V: 04-1] Set up php55lint jobs, adapt phplint macro for $PHP_BIN (re-do) [integration/config] - 10https://gerrit.wikimedia.org/r/268047 (owner: 10Legoktm) [05:15:05] (03PS3) 10Legoktm: Set up php55lint jobs, adapt phplint macro for $PHP_BIN (re-do) [integration/config] - 10https://gerrit.wikimedia.org/r/268047 [06:12:40] hello! [06:12:42] I just realized [06:12:59] that while /data/project is mounted on all deployment-prep hosts [06:13:01] that's not necessary [06:13:04] it can just live on the mw hosts [06:13:10] and varnish if necessary [06:13:17] this will reduce the number of NFS clients we have by a bit [06:13:36] yeah it should only be needed where images are served [06:13:40] and uploaded [06:14:03] (and only until somebody finally sets up swift) [06:14:25] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#1992944 (10yuvipanda) 3NEW [06:14:29] bd808: yeah [06:14:53] bd808: if you (or someone) can clarify where they're actually needed I can setup hiera and whatnot tomorrow [06:15:25] the jobs and mw servers and ... maybe that's it? [06:16:04] I don't remember how the crazy thumbs system is setup in beta cluster [06:16:24] I do remember that it is not like anywhere else on the planet [06:18:55] yeah [06:18:59] it's some nginx somewhere [06:19:16] yeah and an outdated copy of thumb.php [06:19:37] tgr keeps bugging me to help him fix that [06:19:47] :) [06:19:49] thumbor! [06:20:17] yeah there's always some "well we almost have X" excuse ;) [06:21:01] :) [07:17:54] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#1993022 (10yuvipanda) So: # deployment-mediawiki01/02 # deployment-tmh01 # deployment-upload # deployment-cache-upload04 These seem to be the only ones... [07:18:06] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#1993023 (10yuvipanda) a:3yuvipanda [07:42:36] 6Release-Engineering-Team, 3Scap3, 6operations: Depool proxies temporarily while scap is ongoing to avoid taxing those nodes - https://phabricator.wikimedia.org/T125629#1993054 (10jcrespo) 3NEW [08:15:50] (03CR) 10Paladox: [C: 04-1] "Ok I am going to do what hashar suggested todo." [integration/config] - 10https://gerrit.wikimedia.org/r/267911 (owner: 10Paladox) [08:34:58] Yippee, build fixed! [08:34:59] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #864: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/864/ [08:40:24] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [08:54:31] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [09:00:22] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:23:43] can I help? [09:24:31] ps -ef [09:32:40] Hey yurik do you know how to use this room? [09:33:12] hi jeff___ , it depends on what you want to do with it :) [09:35:01] I'm trying to volunteer with wikki. And found this link. [09:35:25] Or room. [09:41:16] PROBLEM - Free space - all mounts on integration-slave-trusty-1016 is CRITICAL: CRITICAL: integration.integration-slave-trusty-1016.diskspace._mnt.byte_percentfree (<42.86%) [09:45:18] gehel that doesn't sound good. u may want to get that looked at! [09:46:28] sound like a bad hard drive. [09:47:11] jeff___: ^^ was that directed to me ? [09:49:28] Yes, I think you have a bad hard drive. [09:51:46] jeff___: what makes you think so ? And which hard drive are you talking about ? (Sorry if I sound lost, I am lost here ...) [09:55:27] I base my infio of what you said "all mounts on integration-slave-trusty-1016 is CRITICAL: CRITICAL: integration.integration-slave-trusty-1016.diskspace._mnt.byte_percentfree (<42.86%) [09:57:12] How dees this irc have anthing to do with wikki? [09:57:32] jeff___: did I say that ? Strange ... [09:58:48] It's cool dude, Talk later, [10:05:30] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #703: 04STILL FAILING in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/703/ [10:06:08] PROBLEM - Free space - all mounts on integration-slave-trusty-1016 is CRITICAL: CRITICAL: integration.integration-slave-trusty-1016.diskspace._mnt.byte_percentfree (<11.11%) [10:31:53] paladox: good morning. Sorry to have skipped/ignored you yesterday. It has been a busy day [10:32:19] hashar: Ok. and good morning too you too. [10:42:27] !log integration-slave-trusty-1016 out of disk space on /mnt ... [10:42:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:46:52] !log integration-slave-trusty-1013 heading to out of disk space on /mnt ... [10:46:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:49:32] !log various beta cluster have puppet errors .. [10:49:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:53:08] !log integration: switched puppet repo back to 'production' branch, rebased. [10:53:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:57:39] (03PS1) 10Adrian Lang: Add me to the Wikidata notification list [integration/config] - 10https://gerrit.wikimedia.org/r/268084 [10:59:15] !log integration/beta: deleting /etc/apt/apt.conf.d/*proxy files. There is no need for them, in fact web proxy is not reachable from labs [10:59:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:07:02] !log beta: apt-get upgrade on delpoyment-cache* hosts and checking puppet [11:07:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:08:59] !log restarting beta cluster puppetmaster just in case [11:09:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:17:15] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [11:19:53] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0] [11:24:43] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [11:26:39] eek [11:32:07] !log fixing puppet.conf on deployment-memc04 [11:32:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:36:55] hashar: Please review https://gerrit.wikimedia.org/r/#/c/267921/ and https://gerrit.wikimedia.org/r/#/c/267624/ and https://gerrit.wikimedia.org/r/#/c/267548/ and https://gerrit.wikimedia.org/r/#/c/267541/ please. [11:37:16] hashar: Also please review https://gerrit.wikimedia.org/r/#/c/267540/ [11:39:52] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [11:40:39] !log apt-get upgrade deployment-ms-be01 and deployment-ms-be02 [11:40:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:45:17] (03CR) 10JanZerebecki: "Deployed to Jenkins: (['browsertests-Wikidata-WikidataTests-linux-chrome-sauce', 'browsertests-Wikidata-WikidataTests-linux-firefox', 'bro" [integration/config] - 10https://gerrit.wikimedia.org/r/268084 (owner: 10Adrian Lang) [11:47:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [11:47:05] (03CR) 10JanZerebecki: [C: 032] Add me to the Wikidata notification list [integration/config] - 10https://gerrit.wikimedia.org/r/268084 (owner: 10Adrian Lang) [11:49:38] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:52] (03Merged) 10jenkins-bot: Add me to the Wikidata notification list [integration/config] - 10https://gerrit.wikimedia.org/r/268084 (owner: 10Adrian Lang) [11:52:17] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:52:59] RECOVERY - Puppet failure on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:43] RECOVERY - Puppet failure on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:41] (03CR) 10JanZerebecki: "Looks good, remaining open question: What does zuul do for 2 matching triggers? Trigger both?" [integration/config] - 10https://gerrit.wikimedia.org/r/267553 (owner: 10Florianschmidtwelzow) [11:59:59] (03CR) 10JanZerebecki: [C: 031] "Yes according to the docs on trigger at http://docs.openstack.org/infra/zuul/zuul.html#pipelines it does." [integration/config] - 10https://gerrit.wikimedia.org/r/267553 (owner: 10Florianschmidtwelzow) [12:12:33] (03PS2) 10JanZerebecki: Whitelist Cenarium [integration/config] - 10https://gerrit.wikimedia.org/r/267612 (owner: 10Legoktm) [12:12:43] (03CR) 10JanZerebecki: [C: 032] Whitelist Cenarium [integration/config] - 10https://gerrit.wikimedia.org/r/267612 (owner: 10Legoktm) [12:13:01] (03CR) 10Hashar: [C: 04-1] "When Zuul receives an event over Gerrit stream-events, it pass it to each of the pipeline which takes their own decisions." [integration/config] - 10https://gerrit.wikimedia.org/r/267553 (owner: 10Florianschmidtwelzow) [12:13:55] (03Merged) 10jenkins-bot: Whitelist Cenarium [integration/config] - 10https://gerrit.wikimedia.org/r/267612 (owner: 10Legoktm) [12:21:25] (03CR) 10JanZerebecki: [C: 04-1] "What is that supposed to fix? I think those match the same language." [integration/config] - 10https://gerrit.wikimedia.org/r/267829 (owner: 10Legoktm) [12:25:56] (03PS2) 10JanZerebecki: Make new DI job live, set old one to non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/268021 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [12:26:24] (03CR) 10JanZerebecki: [C: 032] Make new DI job live, set old one to non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/268021 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [12:27:42] (03Merged) 10jenkins-bot: Make new DI job live, set old one to non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/268021 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [12:38:30] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10Wikidata: 30% of QUnit tests fail on 1.27.0-wmf.12 branch - https://phabricator.wikimedia.org/T125655#1993636 (10hashar) 3NEW a:3hashar [12:42:59] hashar: before I merge it, wanna look at https://gerrit.wikimedia.org/r/#/c/268031/ Use a wrapper parameter-function so jobs can have multiple functions ? [12:43:24] (03CR) 10JanZerebecki: [C: 031] Use a wrapper parameter-function so jobs can have multiple functions [integration/config] - 10https://gerrit.wikimedia.org/r/268031 (https://phabricator.wikimedia.org/T125498) (owner: 10Legoktm) [12:45:45] jzerebecki: neat! [12:46:51] jzerebecki: I was wondering we had any test covering that, seems we have some basic ones at https://gerrit.wikimedia.org/r/#/c/268031/1/tests/test_zuul_ext_dependencies.py,cm [12:46:53] which is even better [12:54:24] jzerebecki: reviewing it [12:59:34] (03CR) 10Hashar: [C: 04-1] "Looks good at first glance. A few nitpick comment inline." (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/268031 (https://phabricator.wikimedia.org/T125498) (owner: 10Legoktm) [13:02:38] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10Wikidata: 30% of QUnit tests fail on 1.27.0-wmf.12 branch - https://phabricator.wikimedia.org/T125655#1993673 (10hashar) The change https://gerrit.wikimedia.org/r/#/c/268096/ pass QUnit. It is a dummy change made against mediawiki/core @wmf/1.27.0-wmf.12... [13:02:45] hashar: would it be ok to just merge https://gerrit.wikimedia.org/r/#/c/268100/ ? [13:02:56] i'm not sure if wmf12 is on tin or anywhere? [13:03:15] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: 30% of QUnit tests fail on 1.27.0-wmf.12 branch when proposing a VE submodule dump - https://phabricator.wikimedia.org/T125655#1993674 (10hashar) [13:03:20] (this will need a scap, before wmf12 is deployed) [13:04:10] aude: I went to sleep when team was still cutting the branch [13:05:27] aude: looks like the branch cutter blindly cut from whatever master is [13:06:33] i added it to swat, just so that we/i am reminded later to ask people about this [13:08:16] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1993684 (10hashar) I have sent the dummy change https://gerrit.wikimedia.org/r/#/c/268096/ against MediaWiki core `1.27.0-wmf.12` and the tests are all fine (except phpcs but that is due to the dummy .php... [13:08:38] (03PS1) 10Aude: Update Wikidata submodule [tools/release] - 10https://gerrit.wikimedia.org/r/268101 [13:08:57] oh [13:09:15] aude: so we bump Wikidata as part of the mediawiki train ? [13:09:20] yeah [13:09:31] so yeah [13:09:41] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1993686 (10aude) wikidata needs https://gerrit.wikimedia.org/r/#/c/268100/ (including scap) and https://gerrit.wikimedia.org/r/#/c/268101/ [13:09:42] +2 the make-wmf-branch [13:09:58] aude: have you filled a bug about it ? Just wondering [13:10:18] but if the submodule was pointing to something (wikidata) cut from master, then the qunit tests should have been fixed [13:11:31] (03CR) 10Hashar: [C: 032] Update Wikidata submodule [tools/release] - 10https://gerrit.wikimedia.org/r/268101 (owner: 10Aude) [13:11:37] thanks [13:11:49] (03CR) 10Hashar: "The submodule bump is https://gerrit.wikimedia.org/r/#/c/268100/" [tools/release] - 10https://gerrit.wikimedia.org/r/268101 (owner: 10Aude) [13:12:13] aude: though if I send a dummy change it pass the qunit tests ( https://gerrit.wikimedia.org/r/#/c/268096/ ) [13:12:22] (03Merged) 10jenkins-bot: Update Wikidata submodule [tools/release] - 10https://gerrit.wikimedia.org/r/268101 (owner: 10Aude) [13:12:24] but maybe the new VisualEditor proposed by James depends on the .12 wikidata [13:12:51] neat [13:13:17] can we add Depends-On in the commit message? [13:13:22] would that actually work? [13:14:23] 6Release-Engineering-Team, 10Fundraising-Backlog: make-wmf-branch should cut wmf/* branches from named extension deployment branches - https://phabricator.wikimedia.org/T113428#1993690 (10JanZerebecki) [13:15:43] aude: I am gonna merge your Wikidata bump [13:15:50] and rebase the VE change [13:15:54] I am just going to tweak your commit message :D [13:16:57] 6Release-Engineering-Team, 10Fundraising-Backlog: make-wmf-branch should cut wmf/* branches from named extension deployment branches - https://phabricator.wikimedia.org/T113428#1993698 (10JanZerebecki) #Release is for any projects releases, using #Release-Engineering-Team instead as there is no project for the... [13:17:20] i don't see wmf12 on mira [13:17:27] so i think we are ok to merge stuff there [13:19:07] aude: I have tweaked the commit message https://gerrit.wikimedia.org/r/268100 to add the prev/new commit one line descriptions [13:19:13] if it looks ok I will +2 [13:19:44] 10Deployment-Systems, 10Fundraising-Backlog: make-wmf-branch should cut wmf/* branches from named extension deployment branches - https://phabricator.wikimedia.org/T113428#1993718 (10JanZerebecki) [13:20:04] 10Deployment-Systems, 10Fundraising-Backlog: make-wmf-branch should cut wmf/* branches from named extension deployment branches - https://phabricator.wikimedia.org/T113428#1664788 (10JanZerebecki) Found a better one. [13:20:13] hashar: looks ok [13:20:23] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1993720 (10hashar) We had Wikidata cut of either the old `1.27.0-wmf.10` or `master` branch. https://gerrit.wikimedia.org/r/#/c/268100/ tweak the submodule to point to Wikidata `1.27.0-wmf.10`. [13:20:32] 10Continuous-Integration-Config, 10Deployment-Systems, 10MediaWiki-extensions-CentralNotice, 10MediaWiki-extensions-Validator, and 2 others: [Task] make core wmf branches only use submodule branches that run with it in CI - https://phabricator.wikimedia.org/T113731#1993721 (10JanZerebecki) [13:20:55] 10Deployment-Systems, 10Wikimedia-General-or-Unknown: Update and rename make-extension-branches script to also handle skin branches - https://phabricator.wikimedia.org/T73753#1993723 (10JanZerebecki) [13:21:35] aude: not sure how that is going to fix the qunit tests [13:21:45] maybe the proposed VE require that Wikidata submodule change [13:22:17] i am not sure also, since wmf.12 wasn't using old code from say 2 weeks ago [13:23:22] hmmm https://gerrit.wikimedia.org/r/#/c/268019/ passes [13:24:18] looks racy [13:30:24] going to grab a snack [13:38:28] thanks :-} [13:39:05] top thing I hate in our workflow [13:39:21] : when we merge a change to an extension, we still have to manually submodule bump :( [13:45:24] Language "zh" failed to load. ... [13:45:53] ok, still failing :( [13:47:25] I have seen that one before [13:47:57] so [13:48:15] i am checking out the core branch [13:48:42] I think we could create wmf branches in mediawiki/extensions.git with the list of extensions we want. Then in MediaWiki wmf branches replace /extensions/ dir to be a submodule checkout of mediawiki/extensions.git [13:48:50] though actually [13:48:50] and blam solved [13:48:50] :D [13:48:51] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/28860/console [13:49:20] eek [13:50:00] [proxy]: failed to proxy /jenkins-mediawiki-extensions-qunit-28860/load.php?skin=fallback&lang=zh&debug=false&modules=mediawiki.language.data%7Cmediawiki.language&only=scripts (browser hung up the socket) [13:50:03] that is nice [13:51:03] merging the VE change [13:51:22] and coffee break [13:52:23] so, it's only one failure? [13:52:29] * aude trying to understand the output [13:55:05] yeah only one [13:55:20] the runner reports the number of success / failure at each test [13:55:56] maybe chromium has some issue on the salve [13:56:05] you get stuff output such as Executed 143 of 1593 SUCCESS (0 secs / 2.553 secs) [13:56:15] VE one is spamming at https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/28893/console [13:57:36] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: 30% of QUnit tests fail on 1.27.0-wmf.12 branch when proposing a VE submodule dump - https://phabricator.wikimedia.org/T125655#1993846 (10hashar) Looks like it was a one time error and then a racy test. The change landed. [13:57:43] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1993848 (10hashar) [13:57:45] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: 30% of QUnit tests fail on 1.27.0-wmf.12 branch when proposing a VE submodule dump - https://phabricator.wikimedia.org/T125655#1993847 (10hashar) 5Open>3Resolved [13:58:01] 10Deployment-Systems, 10Salt: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#1993849 (10ArielGlenn) Actually no addition is needed to the code; there are two sets of data to be removed. One is the key 'deploy::minions:' and its value. The... [14:17:04] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1993908 (10hashar) [14:28:44] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #775: 04FAILURE in 2 min 43 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/775/ [14:35:41] !log manually triggered a bunch of browser tests jobs [14:35:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:38:39] 10Deployment-Systems: make-wmf-branch used master instead of the branch specified in special_extensions - https://phabricator.wikimedia.org/T125663#1993926 (10JanZerebecki) 3NEW [14:38:58] 10Deployment-Systems: make-wmf-branch used master instead of the branch specified in special_extensions - https://phabricator.wikimedia.org/T125663#1993934 (10JanZerebecki) [14:39:57] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1993936 (10Luke081515) [14:40:44] Who wrote "Task to be used to track blockers for the `1.20.0-wmf.12` deployment." at T125475? [14:40:55] do we have Mediawiki 1.20.0 back? [14:40:57] :) [14:41:08] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1988477 (10Luke081515) [14:42:18] !log pooled back integration-slave-trusty-1015 Seems ok [14:42:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:42:50] Luke081515: my mistake I guess :-} [14:43:29] interesting task description :D [14:44:00] I changed it to 1.27.0 [14:44:01] I copied the toc entries from https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys [14:44:03] hoping to help [14:44:07] thank you ! [14:44:11] np [14:46:09] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #888: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/888/ [14:48:16] 10Deployment-Systems, 10Salt: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#1993952 (10ArielGlenn) patchset referenced above has been tested and does the job. I'd like someone else's eyes / hands on it before I roll it into a package. [14:55:05] !log upgrading deployment-stream [14:55:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:58:07] 10Deployment-Systems: make-wmf-branch used master instead of the branch specified in special_extensions - https://phabricator.wikimedia.org/T125663#1993960 (10JanZerebecki) The branch was corrected in https://gerrit.wikimedia.org/r/#/c/268100/ . [15:02:14] Yippee, build fixed! [15:02:15] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #889: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/889/ [15:02:49] 10Deployment-Systems, 10Fundraising-Backlog: make-wmf-branch should cut wmf/* branches from named extension deployment branches - https://phabricator.wikimedia.org/T113428#1993974 (10Krenair) Sounds like we should make CentralNotice use the ordinary setup instead of changing make-wmf-branch. Also, note that w... [15:03:54] !log redeployed rcstream/rcstream on deployment-stream by using git-deploy on deployment-bastion [15:03:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:06:49] hashar: Could you review https://gerrit.wikimedia.org/r/#/c/267548/ and https://gerrit.wikimedia.org/r/#/c/267541/ please. [15:08:13] 10Deployment-Systems, 10Salt, 5Patch-For-Review: trebuchet should expect salt APIs to be asynchronous and poll for status updates from all minions - https://phabricator.wikimedia.org/T103013#1993993 (10ArielGlenn) I've tested the patchset above but on a cluster that's too small and too fast. It does what is... [15:12:40] !log apt-get upgrade deployment-sentry2 [15:12:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:23:27] 10Deployment-Systems: make-wmf-branch used master instead of the branch specified in special_extensions - https://phabricator.wikimedia.org/T125663#1994070 (10Krenair) Isn't this {T113428}? [15:29:26] hashar: [15:29:41] paladox: yeah I have seen your patch [15:29:44] can't review today [15:29:48] oh ok. [15:30:14] I am preparing the mediawiki deployment on the wikimedia cluster [15:31:01] hashar: Oh do you mean your updating the wikis to 1.27 wmf12. [15:31:58] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: 30% of QUnit tests fail on 1.27.0-wmf.12 branch when proposing a VE submodule dump - https://phabricator.wikimedia.org/T125655#1994075 (10Krenair) Are you sure it wasn't just fixed by https://gerrit.wikimedia.org/r/#/c/267892/ ? [15:38:21] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1994078 (10hashar) Clone completed on `mira` ``` Created LocalSettings.php file. Created StartProfiler.php symlink. Created /w/static/1.27.0-wmf.12/skins symlink.... [15:42:51] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1994092 (10hashar) Manually moved back to .10 ``` lrwxrwxrwx 1 hashar wikidev 17 Feb 3 15:39 /srv/mediawiki-staging/php -> php-1.27.0-wmf.10 lrwxrwxrwx 1 hashar w... [15:44:20] paladox: yes. from wmf.10 to wmf.12 [15:44:29] hashar: Ok. [15:46:03] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1994099 (10hashar) [15:47:45] hashar: wmf12 is failing the qunit tests please see https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/28916/consoleFull [15:48:57] we had https://phabricator.wikimedia.org/T125655 [15:51:19] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: 30% of QUnit tests fail on 1.27.0-wmf.12 branch when proposing a VE submodule dump - https://phabricator.wikimedia.org/T125655#1994106 (10hashar) 5Resolved>3Open Yeah the Wikidata bump might have fixed it. There is still a racy test goin... [15:51:22] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1994108 (10hashar) [15:51:26] hashar: Looking at https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/28916/consoleFull it shows wikibase is the problem. [15:51:50] paladox: Language "zh" failed to load. [15:51:50] .. [15:52:03] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: 30% of QUnit tests fail on 1.27.0-wmf.12 branch when proposing a VE submodule dump - https://phabricator.wikimedia.org/T125655#1994110 (10Paladox) Should add the wikibase project since the errors says wikibase is the problem. [15:52:12] hashar: Oh. [15:56:00] ryasmeen|Away: ready? [15:56:06] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: QUnit tests fail on 1.27.0-wmf.12 branch - https://phabricator.wikimedia.org/T125655#1994114 (10hashar) [16:07:21] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: QUnit tests fail on 1.27.0-wmf.12 branch - https://phabricator.wikimedia.org/T125655#1994126 (10hashar) The qunit reports a failure to get zh mediawiki.language.data and mediawiki.language over the API: ``` [proxy]: failed to proxy /jenk... [16:09:07] 10Continuous-Integration-Infrastructure: qunit test is being flaky - https://phabricator.wikimedia.org/T125536#1994138 (10JanZerebecki) Possibly the same as T125484? [16:12:16] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: QUnit tests fail on 1.27.0-wmf.12 branch - https://phabricator.wikimedia.org/T125655#1994143 (10hashar) [16:12:25] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1994147 (10hashar) [16:13:07] 10Continuous-Integration-Infrastructure: qunit test is being flaky - https://phabricator.wikimedia.org/T125536#1994152 (10hashar) [16:13:39] 10Continuous-Integration-Infrastructure: qunit test is being flaky - https://phabricator.wikimedia.org/T125536#1990230 (10hashar) Yeah that is a similar issue. For some reason it takes too long to load the language data :-/ [16:13:56] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1994166 (10JanZerebecki) [16:13:58] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: QUnit tests fail on 1.27.0-wmf.12 branch - https://phabricator.wikimedia.org/T125655#1994160 (10JanZerebecki) 5Open>3Resolved Yea this happened because the fixes for T117886 weren't in that branch yet, because of T125663. I think the erro... [16:16:47] This doesn't look good in the beta cluster logstash -- "Fatal error: Class undefined: Edit-headerAction in /srv/mediawiki/php-master/includes/actions/Action.php on line 99" [16:17:01] That line is in Action::factory [16:17:03] janzerebecki: Hi it seems wikidata is failing on the REL1_25 and is causing flow to fails on the REL1_25 branch. [16:17:09] no idea what the caller might be [16:18:48] jzerebecki: Hi it seems wikidata is failing on the REL1_25 and is causing flow to fails on the REL1_25 branch. [16:18:56] jzerebecki: https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/48658/consoleFull [16:19:07] jzerebecki: https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/48572/consoleFull [16:20:53] paladox: jzerebecki: we should not inject Wikidata in REL1_25 I guess [16:21:17] paladox: I don't know of any users of wikidata with 1.25, I'm pretty sure there is no developer working on it. so what hashar said^^. [16:21:43] gotta commute back home [16:23:46] hashar: jzerebecki: How do we do that. [16:24:09] paladox: gotta hack something in the way extensions are cloned [16:24:21] and drop wikidata whenever REL branch is less than 1_27 [16:24:28] will be an exercise for later, gotta move [16:24:31] hashar: Oh ok. [16:24:58] hashar: Could we just move wikidata to the mwext-Wikidata test. [16:25:15] and then we can block it for the rel1_25 branch. [16:26:04] 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 10Reading-Web, 5Patch-For-Review, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW-Selenium associates wrong SauceLabs job with Jenkins artifact - https://phabricator.wikimedia.org/T105589#1994214 (10zeljkofilipin) This is now solved, right? [16:34:31] jzerebecki: When going to https://github.com/wikimedia/wikidata it says it is empty [16:39:19] paladox: I have no idea what that is. the one on gerrit is mirrored to https://github.com/wikimedia/mediawiki-extensions-Wikidata [16:39:34] oh thanks. [16:49:50] jzerebecki: I think i fixed part of the problem on the REL1_25 branch but now i get this error Wikibase\Test\ApiErrorReporterTest::testDieException with data set "known error code" (exception 'DataValues\IllegalValueException' with message 'ugh!' in /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm/src/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/api/ApiErrorReporterTest.php:104 [16:49:57] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/48665/consoleFull [16:50:31] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1994318 (10greg) a:3demon Other than the incident report, we are now "back to normal" and deploying SWAT deploys and plan to do the train (1.27-wmf.12... [16:51:05] 10Deployment-Systems: make-wmf-branch used master instead of the branch specified in special_extensions - https://phabricator.wikimedia.org/T125663#1994321 (10JanZerebecki) [16:51:26] paladox: re:"move wikidata to the mwext-Wikidata test"; I don't think you understood what mediawiki-extensions-hhvm is used for. [16:52:19] jzerebecki: Its used to test against hhvm. [16:53:34] paladox: what is the difference to mwext-testextension-hhvm? [16:53:54] jzerebecki: Im not sure what you mean. [16:55:48] paladox: there is mwext-testextension-hhvm and there is mediawiki-extensions-hhvm. both test against hhvm. if that where their only reason, why are there two? what does each of them test against hhvm? [16:56:49] jzerebecki: Im not sure why. The extension is using [16:56:49] - name: mediawiki/extensions/Wikidata [16:56:58] test: [16:56:58] - mwext-testextension-hhvm [16:56:58] gate-and-submit: [16:56:58] - mwext-testextension-hhvm [16:57:34] jzerebecki: It also uses extension-gate and thats where test mediawiki-extensions-hhvm is comming from. [17:00:58] 6Release-Engineering-Team, 10MediaWiki-Unit-tests, 10VisualEditor: QUnit tests fail on 1.27.0-wmf.12 branch - https://phabricator.wikimedia.org/T125655#1994366 (10hashar) Thank @JanZerebecki for the clarification. [17:01:39] 6Release-Engineering-Team, 10DBA, 10MediaWiki-Configuration, 6operations, and 3 others: codfw is in read only according to mediawiki - https://phabricator.wikimedia.org/T124795#1994378 (10Krinkle) [17:01:46] hashar: Should we remove mwext-testextension-hhvm test from wikidata since mediawiki-extensions-hhvm is already run on that repo. [17:01:59] 6Release-Engineering-Team, 10DBA, 10MediaWiki-Configuration, 6operations, and 3 others: codfw is in read only according to mediawiki - https://phabricator.wikimedia.org/T124795#1966220 (10Krinkle) [17:03:07] 10Deployment-Systems, 10Fundraising-Backlog: make-wmf-branch should cut wmf/* branches from named extension deployment branches - https://phabricator.wikimedia.org/T113428#1994389 (10JanZerebecki) > Sounds like we should make CentralNotice use the ordinary setup instead of changing make-wmf-branch. That would... [17:04:44] 10Deployment-Systems, 10Fundraising-Backlog: make-wmf-branch should cut wmf/* branches from named extension deployment branches - https://phabricator.wikimedia.org/T113428#1994396 (10JanZerebecki) I think the title of this task is misleading. [17:05:31] 10Deployment-Systems: make-wmf-branch used master instead of the branch specified in special_extensions - https://phabricator.wikimedia.org/T125663#1994408 (10JanZerebecki) This is about a new bug in make-wmf-branch, T113428 is about how to branch CentralNotice. [17:17:34] zeljkof: I see a patch to the UW tests, do those still even work? [17:17:48] MarkTraceur: well, they run :) [17:17:55] But they're broken [17:17:59] I get emails about that, for sure [17:18:05] Maybe I can spend time on that today. [17:18:22] and we had a bug with reporting, so this commit fixes it [17:25:26] greg-g: can you unmoderate my email to releng@? :) [17:25:32] heh, kk [17:25:47] $ listadmin # is the best [17:25:59] approved [17:27:42] greg-g: thanks :D [17:27:50] * YuviPanda awaits ack from someone to kill moar NFS [17:29:44] thcipriani: Er, mentioned in my bullet list / brain dump on https://phabricator.wikimedia.org/T125506#1992199 [17:29:49] But no task to itself [17:30:10] ostriches: ah, knew I saw this somewhere :) [17:34:33] 10Staging, 10releng-201415-Q4: Create staging cluster (tracking) - https://phabricator.wikimedia.org/T88702#1994553 (10mmodell) [17:34:35] 10Staging: Create staging-elastic* (ElasticSearch machines) - https://phabricator.wikimedia.org/T91552#1994552 (10mmodell) [17:34:38] 10Staging, 5Patch-For-Review: Setup staging-tin as deployment host - https://phabricator.wikimedia.org/T88442#1994549 (10mmodell) 5Open>3declined a:3mmodell staging project is no longer [17:37:12] 10Deployment-Systems, 10MediaWiki-Configuration: extension-list should live in the mediawiki branch rather than mediawiki-config - https://phabricator.wikimedia.org/T125678#1994561 (10mmodell) 3NEW a:3mmodell [17:38:57] ostriches: can you review https://gerrit.wikimedia.org/r/#/c/268147 for me plz? MobileFE in the wmf.12 branch is using compsoer.json and that is making wikibase and hhvm sad. [17:39:11] I took a wild stab at fixing but wild and stabby [17:40:19] legoktm: there are currently two merged but undeployed changes to zuul/layout.yaml, one of which is yours. is that ok to deploy? [17:40:34] it's Ib18fe48fe771a3fe381ff4b8c7ee2afb9ebb59e4, a change to the whitelist [17:40:51] marxarelli: he's stil in AU I think so probably not around at the moment [17:41:04] bd808: ah! [17:41:11] well, it seems pretty innocuous [17:41:15] i'll just deploy it [17:42:40] thanks marxarelli -- i will take the demerits if it explodes [17:43:42] !log Reloading Zuul to deploy previously undeployed Icd349069ec53980ece2ce2d8df5ee481ff44d5d0 and Ib18fe48fe771a3fe381ff4b8c7ee2afb9ebb59e4 [17:43:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:44:05] bd808: Oh, that was the thing from yesterday right? [17:44:29] Er, maybe not [17:52:09] marxarelli: worked like a charm! https://gerrit.wikimedia.org/r/#/c/268175/ thanks again [17:52:24] our CI now actually tests the code we deploy [17:52:25] cwd: np! [17:52:33] imagine that! [17:53:05] it's a Feature® [17:56:07] cwd: welcome to the future! [17:56:34] :D [18:06:12] 10Deployment-Systems, 3Scap3: refreshCdbJsonFiles should be rewritten in python - https://phabricator.wikimedia.org/T125685#1994700 (10thcipriani) 3NEW [18:25:17] ostriches: the config change is for something we noticed today after we squashed another similar problem in MFE config during SWAT [18:26:32] Before or during? [18:26:37] If before can do it now [18:30:10] thcipriani: Added T125685 ^ to actionables [18:32:00] 6Release-Engineering-Team, 3Scap3, 6operations: Depool proxies temporarily while scap is ongoing to avoid taxing those nodes - https://phabricator.wikimedia.org/T125629#1994855 (10demon) p:5Triage>3Normal [18:44:00] Krenair: hi, I saw you added me to the subscribers of T125209 but this is a restricted task, I don't have access to it [18:47:03] Kelson, it was a mistake, I meant to add someone else [18:47:16] Krenair: good news ;) [19:03:23] (03CR) 10Florianschmidtwelzow: "Ok, I'll look into it, how to write tests (tests are always good :)), and if I've questions I ping you in irc :P" [integration/config] - 10https://gerrit.wikimedia.org/r/267553 (owner: 10Florianschmidtwelzow) [19:21:04] hasharDinnerTime: If I want to run the tox-tests locally, I'll get an exception, that test-requirements.txt can't be found? :/ But it's there, I'm a beginner in python tests, maybe you can assist? :) [19:24:15] FlorianSW: not this evening sorry. I am pushing a new mw version on wikimedia infra [19:24:15] FlorianSW: folks in #pywikibot should be able to assist or eventually #pypa [19:24:21] else fill it as a task and cc me or add me as a reviewer of the change and I can look it up tomorrow [19:24:23] sorry :-/ [19:25:25] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995084 (10JanZerebecki) [19:25:28] hashar: np :) deploying is a critical task, so I don't want to interrupt you! I'll try it myself, asking in pywikibot and if this all does not help, I'll file a task or ping you in the next days. Thanks for the fast response! [19:28:03] FlorianSW: sounds like a good plan. Thank you for your understanding! [19:28:19] np ;) [19:37:36] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995146 (10hashar) [19:40:47] (03CR) 10JanZerebecki: "Yea triggering both on a +2 is fine. (Wanted to make sure that it wasn't only the test pipeline.)" [integration/config] - 10https://gerrit.wikimedia.org/r/267553 (owner: 10Florianschmidtwelzow) [19:44:13] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995180 (10hashar) [19:45:33] 6Release-Engineering-Team, 3Scap3: updateWikiversions: Don't assume that all versions being operated on +/- of each other - https://phabricator.wikimedia.org/T125702#1995186 (10demon) 3NEW [19:45:48] 6Release-Engineering-Team, 3Scap3: updateWikiversions: Don't assume that all versions being operated on +/- of each other - https://phabricator.wikimedia.org/T125702#1995193 (10demon) Might not be scap project really, it's multiversion code... [19:49:04] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995226 (10demon) [19:49:05] (03PS2) 10Florianschmidtwelzow: Make "recheck" magic word less restrictive for Code-Review votes [integration/config] - 10https://gerrit.wikimedia.org/r/267553 [19:49:46] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 3Collaboration-Team-Current: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#1995227 (10Mattflaschen) >>! In T95871#1648943, @Reedy wrote: > https://github.com/wikimedia/mediawiki/blob/master/maintenance/storage/moveToExternal.p... [19:53:58] 10Continuous-Integration-Config: Running tox on a new environment fails with "No such file or directory" - https://phabricator.wikimedia.org/T125705#1995236 (10Florian) 3NEW [19:56:21] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995256 (10demon) [19:58:56] 10Continuous-Integration-Config: Running tox on a new environment fails with "No such file or directory" - https://phabricator.wikimedia.org/T125705#1995260 (10hashar) Maybe due to `tox 1.6.0`. You could install a newer version without cluttering the /usr/lib with: ``` pip install --user tox ``` Would probably... [19:59:09] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995263 (10demon) [19:59:09] FlorianSW: and I had to reply. nice bug! [19:59:13] FlorianSW: I would upgrade tox [19:59:18] and see if it fix it [19:59:42] hashar: I used the latest version from the ubuntu repository :/ [19:59:50] and, btw., thanks :] [20:00:35] not sure whether it will fix it [20:00:39] would have to look at tox changelog [20:00:51] but that would not surprise if tox 1.6. had trouble with paths [20:02:18] hashar: ok, I'll try it :) [20:05:20] hmm, that seems to work! :) Let me add a notice into the README after testing your second thought, which might be easier in a long term :) [20:07:12] ok, the 2nd suggestion works, too. I'll add a notice to the README, because I think, we don't need to support outdated tox versions, but it should be documented :) [20:08:04] FlorianSW: I think the tox.ini has a minversion = [20:08:23] maybe you can bump it to 1.9.0 [20:08:24] I'll check it :) [20:08:39] propose a patch and attach it to the task [20:08:40] well done! [20:08:55] note it install our fork/snapshot of zuul from git+https://gerrit.wikimedia.org/r/p/integration/zuul.git@debian/precise-wikimedia#egg=zuul [20:09:04] need zuul for a bunch of the tests [20:10:24] hashar: I'll upload it in a second. I saw this while installing the deps manually :) [20:14:18] (03PS1) 10Florianschmidtwelzow: Bump tox minversion to 1.9 [integration/config] - 10https://gerrit.wikimedia.org/r/268207 (https://phabricator.wikimedia.org/T125705) [20:16:10] 10Continuous-Integration-Config, 5Patch-For-Review: Running tox on a new environment fails with "No such file or directory" - https://phabricator.wikimedia.org/T125705#1995316 (10Florian) Running the newer version of tox (installed with `pip install --user tox`) works fine, as well as changing the tox.ini file... [20:16:33] hmm, it's a dependency of the other change now... thanks gerrit :P [20:17:19] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#1995318 (10demon) Logging server? I think we write those syslogs to /data/project [20:17:29] (03PS2) 10Florianschmidtwelzow: Bump tox minversion to 1.9 [integration/config] - 10https://gerrit.wikimedia.org/r/268207 (https://phabricator.wikimedia.org/T125705) [20:17:39] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995321 (10hashar) [20:18:09] (03CR) 10Florianschmidtwelzow: "PS2 removes the un-intentional dependency to I2e0ea1069c40bc539fe848abf0919b602613f746" [integration/config] - 10https://gerrit.wikimedia.org/r/268207 (https://phabricator.wikimedia.org/T125705) (owner: 10Florianschmidtwelzow) [20:18:40] (03PS3) 10Florianschmidtwelzow: Bump tox minversion to 1.9 [integration/config] - 10https://gerrit.wikimedia.org/r/268207 (https://phabricator.wikimedia.org/T125705) [20:38:34] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [20:45:39] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1995411 (10phuedx) [21:05:14] hashar: Would you know off any temporary soloutions for wikidata on branch REL1_25. [21:24:06] 10Beta-Cluster-Infrastructure, 6Labs: Disable /data/project for instances in deployment-prep that do not need it - https://phabricator.wikimedia.org/T125624#1995517 (10yuvipanda) I am pretty sure we killed that when we found out. [21:26:59] 10Deployment-Systems, 10Fundraising-Backlog: make-wmf-branch doesn't work for named extension deployment branches - https://phabricator.wikimedia.org/T113428#1995527 (10awight) [21:27:46] paladox: we will want some dirty hack in the zuul-cloner macro that has wikidata so the extension is not added for REL1_25 / REL1_26 [21:28:01] paladox: doing a long tedious deployment tonight so can't really look at it [21:28:03] :( [21:28:32] hashar: Ok. Are you deploying it mediawiki 1.27 wmf12 to all wikimedia wikis. [21:29:04] to testwiki for now [21:29:12] main IRC channel is #wikimedia-operations [21:29:17] hashar: Do you know of anyother repos that have a hack like the one we are trying todo with wikidata. Since i could have alook at it. [21:29:41] ok. So once testwiki is successfull you will move onto the next wiki. [21:29:41] I dont think we have any [21:29:46] Oh ok. [21:30:23] ostriches: I'm pretty sure I killed syslog to NFS when I found out many months ago [21:30:42] Ok nvm then [21:31:04] ostriches: can you +1 then? :D [21:32:03] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995553 (10hashar) [21:34:10] 10Deployment-Systems, 10Fundraising-Backlog: make-wmf-branch doesn't work for named extension deployment branches - https://phabricator.wikimedia.org/T113428#1995556 (10awight) Thank you both for taking a look at this! It's very possible that I don't understand how release branching works at the WMF, and I'd... [21:37:39] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1995591 (10greg) 5Resolved>3Open This is back: 21:36 < hashar> mira has l10nupdate uid == 10002 tin has l10nupdate uid = 1001 [21:37:42] 10Deployment-Systems, 3Scap3, 5Patch-For-Review, 7WorkType-NewFunctionality: [scap] Add support for syncing /srv/mediawiki-staging including fully working git data to warm spare deploy server - https://phabricator.wikimedia.org/T104826#1995593 (10greg) [21:38:01] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1995594 (10hashar) Following reinstallation of tin on Feb 2nd: | mira | uid=10002(l10nupdate) gid=10002(l10nupdate) groups=10002(l10nupdate) | tin | uid=1001(l1... [21:38:23] greg-g: Do you know if disabled Phab account receive notifications? [21:38:29] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1995597 (10hashar) [21:38:30] I don't know [21:38:31] (I'm hoping not) [21:38:32] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995596 (10hashar) [21:38:32] not sure [21:38:38] twentyafterfour: ^ [21:38:42] re phab question [21:38:59] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1988477 (10hashar) The `l10nupdate` user UID mismatch between tin and mira preventing sync-masters to push the l10n cache files Reopened T119165 [21:39:04] Otherwise we purge them from subscriber lists too, for security reasons and just in general. [21:39:11] (for e..g. former wmfers) [21:41:11] Krinkle: they shouldn't... though I think ostriches might have said that they do ... [21:42:06] They don't receive most notifs. [21:42:34] wasn't there an issue with email sent to disabled accounts? [21:42:35] They *do* still get their "confirm your e-mail" notif if they're unlucky enough to try and confirm prior to being locked. [21:42:42] ah [21:43:11] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1995631 (10hashar) So @ori had a patch https://gerrit.wikimedia.org/r/#/c/256026/4/modules/scap/manifests/l10nupdate.pp,cm but that does not show up anymore in... [21:51:31] ostriches: But if I remove a disabled user account from the subscirber list that won't notif them? [21:51:43] It shouldn't. [21:52:07] If they're disabled they shouldn't get any notifs (other than the 'please confirm yer email' notif like I said) [21:52:20] In the case of former wmfers, those will all bounce anyway :p [21:59:02] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1995688 (10Dzahn) I went to tin and adjusted the UID of the l10nupdate user, after confirming 10002 is correct on https://wikitech.wikimedia.org/wiki/UID `vi /e... [22:03:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 10Wikidata: Wikidata broken on branch REL1_25 causing other extensions to fail. - https://phabricator.wikimedia.org/T125722#1995710 (10Paladox) 3NEW [22:04:09] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 10Wikidata: Wikidata broken on branch REL1_25 causing other extensions to fail. - https://phabricator.wikimedia.org/T125722#1995722 (10Paladox) @JanZerebecki can I add you to task please. [22:04:14] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 10Wikidata: Wikidata broken on branch REL1_25 causing other extensions to fail. - https://phabricator.wikimedia.org/T125722#1995724 (10Paladox) [22:04:23] hashar: Ive opened the task at https://phabricator.wikimedia.org/T125722 [22:05:38] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1995731 (10Dzahn) same on mira (and tin) for: `find /srv/mediawiki-staging/ -uid 1001 -exec chown 10002 {} \;` [22:10:45] hashar: I think a temporary solotions is to disable the tests in wikidata branch REL1_25. Meaning not disable test in jenins but you disable them in one of the files. [22:12:32] 7Browser-Tests, 10MediaWiki-Authentication-and-authorization: Create some end-to-end tests for SessionManager - https://phabricator.wikimedia.org/T125599#1995768 (10Anomie) a:5Anomie>3dduvall [22:14:55] 10Deployment-Systems, 3Scap3, 5Patch-For-Review, 7WorkType-NewFunctionality: [scap] Add support for syncing /srv/mediawiki-staging including fully working git data to warm spare deploy server - https://phabricator.wikimedia.org/T104826#1995785 (10Dzahn) [22:14:57] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1995783 (10Dzahn) 5Open>3Resolved [22:16:14] 10Deployment-Systems, 6operations, 5Patch-For-Review: l10nupdate user uid mismatch between tin and mira - https://phabricator.wikimedia.org/T119165#1819881 (10Dzahn) the immediate blocker is fixed. for the puppetization issue see comments on https://gerrit.wikimedia.org/r/#/c/255421/ [22:19:52] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1995810 (10hashar) Got .12 on `testwiki` [22:24:35] !log Manually ran sync-common on deployment-jobrunner01.eqiad.wmflabs to pickup wmf-config changes that were missing (InitializeSettings, Wikibase, mobile) [22:24:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:26:35] jzerebecki: Could you merge https://gerrit.wikimedia.org/r/#/c/268306/ please. [22:28:44] addshore: Could you review https://gerrit.wikimedia.org/r/#/c/268306/ please. [22:29:55] 10Deployment-Systems, 10Salt, 6operations, 5Patch-For-Review: [Trebuchet] Salt times out on parsoid restarts - https://phabricator.wikimedia.org/T63882#1995861 (10ArielGlenn) [22:34:05] marxarelli: err, I didn't merge it o.O [22:34:23] paladox: just had them as reviewer I believe [22:35:09] hashar: Ok. [22:36:07] legoktm: oh sorry, my mistake. in any case, it's deployed :) [22:39:45] thanks :) [22:40:05] 10Deployment-Systems, 10Salt, 6operations, 5Patch-For-Review: [Trebuchet] Salt times out on parsoid restarts - https://phabricator.wikimedia.org/T63882#1995902 (10ArielGlenn) (12:32:36 πμ) subbu: ssastry@mira:/srv/deployment/parsoid/deploy$ git deploy service restart (12:32:36 πμ) subbu: Error received fro... [22:51:25] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1995927 (10hashar) 3NEW a:3hashar [22:52:01] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1995935 (10Paladox) [22:53:51] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1995942 (10bd808) Then I use the `X-Wikimedia-Debug: 1` header to route my requests to mw1017 performance seems normal. Without the header I'm seeing lo... [22:59:09] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1995979 (10Florian) Just to confirm: happens for me, too, I jsut wanted to test something (;)) and I was wondering why all actions (rendering of existin... [23:02:36] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 3Collaboration-Team-Current: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#1995991 (10jcrespo) > In my opinion should be enough to switch it then create some content on Beta (the browser tests should also help with that). I w... [23:09:26] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #407: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/407/ [23:11:36] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1996036 (10hashar) testwiki / testwiki2 are on wmf12 but they are terribly slow {T125727} I am giving up myself (past midnight here) and pass relay to @thcipriani... [23:17:50] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising-CiviCRM: CI for Civi: provision and run tests under Jenkins/Zuul - https://phabricator.wikimedia.org/T86103#1996090 (10Eileenmcnaughton) [23:20:37] 10Deployment-Systems, 10MediaWiki-Configuration: extension-list should live in the mediawiki branch rather than mediawiki-config - https://phabricator.wikimedia.org/T125678#1996104 (10mmodell) [23:21:18] we're redesigning the release process over in #wikimedia-tech. hope you don't mind ;-) [23:29:41] 6Release-Engineering-Team, 5Patch-For-Review, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1996176 (10bd808) Page generation times seem to be getting better and better for testwiki. @ori had a reasonable explanation that t... [23:33:28] 6Release-Engineering-Team, 5Patch-For-Review, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1996211 (10hashar) p:5Unbreak!>3Low **note** we want to update our deployment train doc to have the whole group0 switched inste... [23:38:18] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1996245 (10greg) [23:38:20] 6Release-Engineering-Team, 5Patch-For-Review, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1996243 (10greg) 5Open>3Invalid This was from expect HHVM cache warming. Full IRC log below for those who weren't there: ```... [23:40:13] 6Release-Engineering-Team, 5Patch-For-Review, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): test / test2 incredibly slow with 1.27.0-wmf.12 - https://phabricator.wikimedia.org/T125727#1996253 (10thcipriani) 5Invalid>3Resolved 1.27.0-wmf.12 is now out on all group0 wikis and seems to be moving at a normal clip.... [23:40:15] 6Release-Engineering-Team, 5WMF-deploy-2016-02-02_(1.27.0-wmf.12): MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1996255 (10thcipriani) [23:40:45] heh, edit collision [23:45:08] So, are we going to put wmf.12 on group1 now? [23:45:44] https://phabricator.wikimedia.org/T125484 is still marked as a blocker. [23:47:21] yes, we will, and yes, I think that's wrong (it being a full blocker) for now [23:47:30] sadly the workaround is increasing the timeout on those jobs