[00:01:32] hmm, why is https://gerrit.wikimedia.org/r/#/c/248526/ looks older version in beta than in prod? [00:01:52] i.e. the content of submodule is old [00:03:27] maybe a change was merged in the submodule but there was no matching change in the parent module to update the submodule [00:03:34] submodules always create trouble like that [00:04:24] well, it somehow worked in prod... [00:04:58] beta uses a different puppetmaster [00:05:02] yup [00:05:23] aaaaargh [00:05:27] drwxr-sr-x 3 smalyshev wikidev 4096 Oct 15 22:11 portals [00:05:46] and zuul dares to claim the update worked! [00:07:27] MaxSem, is it really set up to deal with submodules? [00:07:48] FFOS app allegedly works [00:08:02] or just nobody cared to check [00:08:23] My assumption would be that nobody has checked [00:08:33] is this still about the portal change? [00:08:40] yes [00:08:43] but yeah, the problem is that Stas did some stuff manually that broke teh automatic stuff [00:08:46] that is an unfortunate combo given the controversial nature of this change [00:09:38] it's not controversial... yet [00:09:50] controversial would be to point apaches to it [00:09:57] oh, there was lots of discussion on gerrit and phab [00:09:58] Didn't they do that? [00:10:09] only in beta [00:10:22] to, you know, check how it works [00:10:46] * MaxSem imagines the nice crater that woud've appeared otherwise [00:12:39] !log Manually fixed permissions on mw-config/portals, reinitialized submodule and synced [00:12:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:13:14] I expect you're going to have to have an extra deployment step for these portals to update beta, MaxSem [00:13:27] we'll see [00:13:31] Still, maybe it's a good thing to update beta first. [00:14:05] essentially, this is gonna be a simple git submodule update after git pull [00:14:21] Which isn't necessarily going to get automatic updates. [00:14:44] okay, progress: http://www.wikipedia.beta.wmflabs.org/ is a 404, not 503 [02:00:20] 10Beta-Cluster-Infrastructure: +Sysop for User:Mww113 - https://phabricator.wikimedia.org/T116364#1764602 (10Mww113) Enwiki account created [02:40:51] !log beta-scap-eqiad failing due to rsync-created mira:/srv/mediawiki-staging/.~tmp~ directory being owned by mwdeploy but with a uid of 993 instead of 603 (local mwdeploy) [02:40:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:49:56] Yippee, build fixed! [02:49:56] Project beta-scap-eqiad build #76335: 09FIXED in 5 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76335/ [02:50:16] !log hooray, fixed! [02:50:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:01:32] Project beta-scap-eqiad build #76336: 04FAILURE in 7 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76336/ [03:17:25] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #763: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/763/ [03:18:06] !log broken again. looks like /srv/mediawiki-staging on mira should be owned by mwdeploy [03:18:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:26:20] Yippee, build fixed! [03:26:21] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #863: 09FIXED in 44 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/863/ [03:27:00] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1764651 (10thcipriani) 3NEW [06:26:57] Yippee, build fixed! [06:26:58] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #789: 09FIXED in 7 min 57 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/789/ [06:32:36] PROBLEM - Puppet failure on integration-slave-trusty-1023 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:39:01] 10Beta-Cluster-Infrastructure: +Sysop for User:Mww113 - https://phabricator.wikimedia.org/T116364#1764761 (10Luke081515) 5Open>3Resolved Done [06:57:10] PROBLEM - Puppet failure on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:07:35] RECOVERY - Puppet failure on integration-slave-trusty-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:07] RECOVERY - Puppet failure on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:15] 10Continuous-Integration-Config, 6Collaboration-Team-Backlog, 10Echo: Generate doxygen for Echo on doc.wm.o - https://phabricator.wikimedia.org/T117025#1764815 (10Legoktm) 3NEW [07:39:24] 10Continuous-Integration-Config, 6Collaboration-Team-Backlog, 10Echo: Generate doxygen for Echo on doc.wm.o - https://phabricator.wikimedia.org/T117025#1764823 (10Legoktm) p:5Triage>3Low [07:56:58] 10Continuous-Integration-Infrastructure: Migrate all jobs to labs slaves - https://phabricator.wikimedia.org/T86659#1764837 (10Legoktm) I suppose we could turn the `integration-docroot-deploy` into a rsync job like the doc ones? Is it worth it? I'm guessing `mediawiki-vagrant-puppet-doc` is straightforward, sim... [09:26:03] 5Continuous-Integration-Scaling, 6operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1764875 (10hashar) Thanks for all the attention. The pbuider hook defines `jessie-wikimedia/backports` but the packages are in upstream `jessie-... [09:52:54] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-fluorine fails puppet '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists - https://phabricator.wikimedia.org/T117028#1764903 (10hashar) 3NEW [10:21:10] !log restarting Jenkins (java upgrade) [10:21:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:58:27] 10Browser-Tests, 7Easy, 7Tracking: Remove regular expressions in browser tests page element selectors - https://phabricator.wikimedia.org/T112291#1764957 (10TTO) [10:58:50] 10Browser-Tests, 7Easy, 7Tracking: Remove regular expressions in browser tests page element selectors - https://phabricator.wikimedia.org/T112291#1631130 (10TTO) Seems like this is not blocking the release of 1.26, as end users won't be running tests. Please revert me if I'm wrong though! [11:02:38] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 10Unplanned-Sprint-Work, 10Wikidata, and 3 others: ContentTranslation zend tests fail because of Wikibase\Test\EditEntityTest::testAttemptSaveRateLimit - https://phabricator.wikimedia.org/T116732#1764963 (10JanZerebecki) 5Open>3... [11:21:17] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:28:06] PROBLEM - Puppet failure on deployment-sca02 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [12:56:21] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:04] RECOVERY - Puppet failure on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:43:21] 5Continuous-Integration-Scaling, 7Tracking: Investigate using a cache store/restore system for package managers - https://phabricator.wikimedia.org/T116017#1765169 (10JanZerebecki) As long as we trust our +2ers, using one cache per type (i.e. one for composer, different one for gem) should be fine. Another sol... [14:28:20] Yippee, build fixed! [14:28:21] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #673: 09FIXED in 2 min 19 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/673/ [14:35:52] Yippee, build fixed! [14:35:52] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #307: 09FIXED in 7 min 50 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/307/ [14:45:34] Yippee, build fixed! [14:45:34] Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #52: 09FIXED in 4 min 23 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/52/ [15:07:31] 10Beta-Cluster-Infrastructure, 6operations, 7WorkType-NewFunctionality: etcd/confd is not started on deployment-cache-mobile04 - https://phabricator.wikimedia.org/T116224#1765358 (10hashar) ``` deployment-cache-text04:/etc# cat /lib/systemd/system/confd.service [Unit] Description=confd [Service] User=root E... [15:08:54] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1765361 (10hashar) [15:09:47] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1765366 (10hashar) p:5Triage>3Unbreak! That breaks beta-scap-eqiad on deployment-bastion: ``` 15:07:15 15:07:08 Started rsy... [15:11:22] 10Continuous-Integration-Config: Sometimes when c+2 it merges straight away before jenkins has a chance to test - https://phabricator.wikimedia.org/T117046#1765368 (10Paladox) 3NEW [15:11:35] PROBLEM - Puppet failure on deployment-conf03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:11:40] 10Continuous-Integration-Config: Sometimes when c+2 it merges straight away before jenkins has a chance to test - https://phabricator.wikimedia.org/T117046#1765375 (10Paladox) @Hashar and @Legoktm can I add you to task. [15:11:46] 10Continuous-Integration-Config: Sometimes when c+2 it merges straight away before jenkins has a chance to test - https://phabricator.wikimedia.org/T117046#1765377 (10Paladox) [15:19:04] Yippee, build fixed! [15:19:12] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #224: 09FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/224/ [15:19:27] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1765406 (10Krenair) Cherry-picked patch on deployment-puppetmaster, ran puppet on deployment-bastion: `Notice: /Stage[main]/Sca... [15:22:27] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1765423 (10thcipriani) Hmm, mira:`/srv/mediawiki-staging/.~tmp~` is owned by 993 again: thcipriani@mira:/srv/mediawiki-sta... [15:23:06] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1765425 (10hashar) ``` $ stat /srv/mediawiki-staging File: `/srv/mediawiki-staging' Size: 4096 Blocks: 8 IO... [15:26:35] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1765430 (10bd808) >>! In T117016#1765423, @thcipriani wrote: > Hmm, mira:`/srv/mediawiki-staging/.~tmp~` is owned by 993 again:... [15:27:28] thcipriani: that mwdeploy / /srv/mediawiki-staging is crazy [15:27:37] yup, I'm vipw-ing now [15:27:40] !log removed shadow mwdeploy account from /etc/passwd on mira.deployment-prep.eqiad.wmflabs [15:27:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:27:47] oh men [15:27:58] * bd808 is faster [15:28:07] also faster at typing that [15:28:24] I forgot to check the ldap/local stuff [15:28:41] can we abandon puppet? [15:28:43] we haven't seen it for a while (or at least I haven't) [15:28:49] Yippee, build fixed! [15:28:49] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #701: 09FIXED in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/701/ [15:28:58] so [15:29:00] it's not puppet's fault that our ldap system is shit [15:29:00] I don't get it [15:29:03] hashar: please review https://gerrit.wikimedia.org/r/#/c/249418/ and https://gerrit.wikimedia.org/r/#/c/249416/ and https://gerrit.wikimedia.org/r/#/c/248899/ please [15:29:11] is mwdeploy user supposed to be part of the wikidev group ? [15:29:16] paladox: if you added me as a reviewer it is in my review queue [15:29:22] Oh ok. [15:29:39] which is currently ~ 2 thousands emails [15:29:45] hashar, no. mwdeploy can't be a member of wikidev [15:29:52] paladox: so it will take 2 - 3 business days for me to reach them :/ [15:30:01] ok. [15:30:36] (03CR) 10Paladox: "Test failures are because this patch depends on a change to be merged before the test here passes." [integration/config] - 10https://gerrit.wikimedia.org/r/248899 (owner: 10Paladox) [15:30:47] holy hell [15:30:59] bd808: how does deployment-bastion:/srv/mediawiki-staging now belongs to mwdeploy ? [15:31:11] a new puppet rule [15:31:34] RECOVERY - Puppet failure on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:31:51] ho [15:31:54] when we rsync from master to master the mtime of that dir will need to be set and you can only set mtime of a directory if you are the direct owner [15:31:58] it's weirdness [15:32:00] Krenair patch, I missed it [15:32:04] *nod* [15:32:46] the rsync uses a sudoers rule that lets mwdeploy sudo as group wikidev but only to do that one rsync [15:32:54] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira permissions problem - https://phabricator.wikimedia.org/T117016#1765455 (10hashar) https://gerrit.wikimedia.org/r/#/c/249684/ has been cherry picked and /srv/mediawiki-staging now belong to m... [15:33:22] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira / deployment-bastion permissions problem - https://phabricator.wikimedia.org/T117016#1765462 (10hashar) [15:33:42] bd808: the thing is the sudo rule apparently got removed [15:33:54] no it didn't [15:34:21] chad had just masked the problem that Krenair's patch fixed my manually chmoding at some point [15:34:47] I am referring to that rule: mwdeploy ALL = (mwdeploy:wikidev) NOPASSWD: /usr/bin/rsync *\:\:common /srv/mediawiki-staging [15:35:15] no clue where it came from [15:35:23] wasn't that in someone's puppet patch? [15:35:24] mira.deployment-prep:/etc/sudoers.d/scap-master-sync [15:35:35] bd808's scap sync-masters config patch? [15:36:06] It comes from this patch -- https://gerrit.wikimedia.org/r/#/c/224829/ -- which is cherry-picked [15:36:15] it did get moved to a new file [15:36:20] blerg, I was looking for the rule in /etc/sudoers.d/mwdeploy [15:36:27] so that it would only be provisioned on scap masters [15:37:02] Krenair's patch was the missing bit. We could fold them into one patch I suppose [15:37:24] certainly we should link them toghether in gerrit [15:37:56] Yippee, build fixed! [15:37:57] Project beta-scap-eqiad build #76417: 09FIXED in 3 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/76417/ [15:38:10] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira / deployment-bastion permissions problem - https://phabricator.wikimedia.org/T117016#1765478 (10hashar) From @bd808 the mysterious sudo rule comes from a cherry picked patch https://gerrit.wi... [15:38:52] sudo::user { 'scap-master-sync': [15:38:53] grrbpbpb [15:39:24] thanks bd808 :) [15:41:43] (03CR) 10Hashar: [C: 04-1] "Sounds good. While at it can you update all the projects and switch them to use the template? Thx!" [integration/config] - 10https://gerrit.wikimedia.org/r/249416 (owner: 10Paladox) [15:43:03] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira / deployment-bastion permissions problem - https://phabricator.wikimedia.org/T117016#1765501 (10bd808) This is all related to {T104826}. @demon tested things out in the last couple of days, b... [15:46:30] (03CR) 10Hashar: [C: 04-1] [OOUIPlayground] Update Jenkins tests (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/248899 (owner: 10Paladox) [15:47:24] mira is a terrible name for the beta instance [15:47:29] to easy to confuse it with the prod host [15:47:31] [15:48:16] you guys have a mira in beta that is not a deploy host? [15:48:58] no [15:52:00] chasemp: labs is only eqiad so ;) [15:52:48] so we have mira.eqiad.wmflabs and mira.codfw.wmnet [15:53:00] a mistake was made while setting it up, it should have been deployment-mira.eqiad.wmflabs [15:53:08] ah [15:54:03] oh [15:54:11] so prod has two mira ? :( [15:54:46] I thought we went off named hosts in favor of something like such as bast1001 [15:54:51] but that is all nitpicking [15:54:59] at least all three hosts are consistents this way [15:55:55] hashar: do you really want deploy1001 vs tin? :) [15:56:16] They're misc services which follow dc naming schemes [15:56:56] JohnFLewis: I am just ranting / mumbling ;-} [15:57:31] hashar: still, deploy1001.eqiad.wmnet :) [15:59:22] 10Continuous-Integration-Config: Sometimes when c+2 it merges straight away before jenkins has a chance to test - https://phabricator.wikimedia.org/T117046#1765571 (10Legoktm) You're pressing "publish and submit" instead of "publish". [16:01:56] 10Continuous-Integration-Config: Sometimes when c+2 it merges straight away before jenkins has a chance to test - https://phabricator.wikimedia.org/T117046#1765576 (10Paladox) Oh so i am to click publish instead of publish and submit. [16:03:13] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:16:47] (03CR) 10Paladox: "Hashar: Yes it is the correct link I am adding the new template that is required here." [integration/config] - 10https://gerrit.wikimedia.org/r/248899 (owner: 10Paladox) [16:18:25] (03CR) 10Paladox: "The link in commit msg is the correctly link this patch depends on that being merged first." [integration/config] - 10https://gerrit.wikimedia.org/r/248899 (owner: 10Paladox) [16:22:52] (03PS2) 10Paladox: Add new jsduck template [integration/config] - 10https://gerrit.wikimedia.org/r/249416 [16:23:23] (03CR) 10Paladox: "Ok done." [integration/config] - 10https://gerrit.wikimedia.org/r/249416 (owner: 10Paladox) [16:27:00] PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145) [16:27:36] (03PS1) 10Paladox: [MultimediaViewer] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/249763 [16:30:31] 10Continuous-Integration-Config: Sometimes when c+2 it merges straight away before jenkins has a chance to test - https://phabricator.wikimedia.org/T117046#1765651 (10Legoktm) 5Open>3Invalid a:3Legoktm Yes. [16:39:56] (03PS1) 10Legoktm: Add npm for EducationProgram, FeaturedFeeds, GWToolset & Listings [integration/config] - 10https://gerrit.wikimedia.org/r/249766 [16:41:55] (03CR) 10Legoktm: [C: 032] Add npm for EducationProgram, FeaturedFeeds, GWToolset & Listings [integration/config] - 10https://gerrit.wikimedia.org/r/249766 (owner: 10Legoktm) [16:42:53] (03Merged) 10jenkins-bot: Add npm for EducationProgram, FeaturedFeeds, GWToolset & Listings [integration/config] - 10https://gerrit.wikimedia.org/r/249766 (owner: 10Legoktm) [16:50:48] !log deploying https://gerrit.wikimedia.org/r/249766 [16:51:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:53:24] 10Differential, 3releng-201516-q2: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765720 (10mmodell) 3NEW a:3mmodell [16:56:24] 10Differential, 5Gerrit-Migration, 3releng-201516-q2: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765728 (10hashar) [17:00:50] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MediaWiki-extensions-General-or-Unknown: Add grunt-concurrent to mediawiki/core and decide whether to add it to extensions (Improve grunt performance) - https://phabricator.wikimedia.org/T116988#1765752 (10Paladox) [17:04:59] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MediaWiki-extensions-General-or-Unknown: Add grunt-concurrent to mediawiki/core and decide whether to add it to extensions (Improve grunt performance) - https://phabricator.wikimedia.org/T116988#1765765 (10Legoktm) This bug makes sense to me... [17:05:29] 3releng-201516-q2: [keyresult] Deprecate gitblit in favor of Diffusion - https://phabricator.wikimedia.org/T111465#1765767 (10greg) [17:10:38] 10Deployment-Systems, 3Scap3: Scap3 targets should use a config file rather than `key:value` arguments - https://phabricator.wikimedia.org/T116432#1765788 (10dduvall) 5Open>3Resolved [17:11:06] Graying out on Deployments page is so helpful. Thank you whoever did that. [17:12:42] Timo [17:13:24] 10Differential, 5Gerrit-Migration: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765796 (10greg) [17:16:04] 10Continuous-Integration-Config, 10Fundraising-Backlog: Forced merges are breaking Zuul - https://phabricator.wikimedia.org/T117062#1765798 (10awight) 3NEW [17:23:49] greg-g, twentyafterfour is there a process to get changes like https://gerrit.wikimedia.org/r/#/c/249223/1 merged? (Third party, non-wmf-deployed extension related changes for CI)? Maybe anyone I can/should add as a reviewer? :/ [17:24:46] 10Continuous-Integration-Config, 10Fundraising-Backlog: Forced merges are breaking Zuul - https://phabricator.wikimedia.org/T117062#1765837 (10Legoktm) The "something in zuul falls over dead" is {T93812} for reference. [17:26:01] FlorianSW: I'm not sure, hashar would be the one who knows more, I think [17:26:05] he's offline atm [17:26:54] he was the person I was looking for, but yeah, he's offline, so I thought maybe one of your know that, too :) Thanks for the answer, then I will see, if I get him somewhere :) [17:27:41] I will let you know when I see hashar around here [17:27:41] @notify hashar [17:27:52] (03PS2) 10Legoktm: Add GoogleAPIClient as dependency to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/249223 (owner: 10Florianschmidtwelzow) [17:27:56] (03CR) 10Legoktm: [C: 032] Add GoogleAPIClient as dependency to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/249223 (owner: 10Florianschmidtwelzow) [17:28:02] hi FlorianSW :) [17:28:15] hi legoktm :) How are you? [17:29:10] awake somehow. [17:29:46] that sounds good ;) :P [17:31:08] legoktm: ahh, thanks for merging :] I assume you reload zuul, too? :) [17:31:30] yeah, just waiting on jenkins... [17:32:18] +1! :) thanks [17:32:45] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10MediaWiki-extensions-General-or-Unknown: Add grunt-concurrent to mediawiki/core and decide whether to add it to extensions (Improve grunt performance) - https://phabricator.wikimedia.org/T116988#1765854 (10Paladox) @Legoktm thanks for answeri... [17:33:09] 10Differential, 5Gerrit-Migration: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765857 (10mmodell) Repository access controls: {F2892715} {F2892722} [17:33:25] (03Merged) 10jenkins-bot: Add GoogleAPIClient as dependency to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/249223 (owner: 10Florianschmidtwelzow) [17:33:34] !log deploying https://gerrit.wikimedia.org/r/249223 [17:33:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:33:52] 10Continuous-Integration-Config, 7I18n: Configure banana checker for i18n files to run on all MediaWiki extensions and skins (tracking) - https://phabricator.wikimedia.org/T94547#1765863 (10Umherirrender) [17:34:12] FlorianSW: deployed [17:34:22] legoktm: great, thanks :) [17:38:04] FlorianSW: feel free to add me as a reviewer to your CI patches in the future / ping me on IRC about them [17:39:24] legoktm: ok, will do that! (There aren't so many) :) [18:01:34] 10Differential, 5Gerrit-Migration: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765934 (10mmodell) some discussion about differential "sticky accept" T164 [18:04:25] 10Differential, 5Gerrit-Migration, 15User-greg: Uploading a new Differential Revision does not reset “Accept” states in commit messages - https://phabricator.wikimedia.org/T164#1765946 (10mmodell) @scfc: I agree with you that everyone makes silly mistakes and code review is helpful at catching those. I don'... [18:05:29] 5Gerrit-Migration, 7Epic: Align Wikimedia and Phabricator code review processes - https://phabricator.wikimedia.org/T167#1765949 (10mmodell) [18:05:30] 10Differential, 5Gerrit-Migration: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765948 (10mmodell) [18:05:47] 10Differential, 5Gerrit-Migration: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765720 (10mmodell) [18:05:48] 10Differential, 5Gerrit-Migration, 15User-greg: Uploading a new Differential Revision does not reset “Accept” states in commit messages - https://phabricator.wikimedia.org/T164#1765951 (10mmodell) [18:06:19] 10Differential, 5Gerrit-Migration: Support automatic commit generation in Phabricator - https://phabricator.wikimedia.org/T182#1765954 (10mmodell) [18:06:21] 10Differential, 5Gerrit-Migration: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765720 (10mmodell) [18:06:55] 10Differential, 5Gerrit-Migration: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1765720 (10mmodell) [18:06:56] 10Differential, 5Gerrit-Migration: Define an equivalent to Gerrit's +-1 +-2 for code review evaluation - https://phabricator.wikimedia.org/T138#1765956 (10mmodell) [18:19:32] twentyafterfour: ostriches has: "Qgil moved this task to On track on the Wikimedia-Developer-Summit-2016 workboard. [18:19:37] \o/ [18:29:37] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:44:38] 5Gerrit-Migration, 6Labs: Figure out a git hosting solution for tools - https://phabricator.wikimedia.org/T117071#1766048 (10yuvipanda) 3NEW [18:51:29] 5Continuous-Integration-Scaling, 6operations, 10ops-eqiad: Reclaim SSD from labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T116936#1766074 (10Cmjohnson) @chasemp ping when it's a good time to do this. [19:09:43] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:14:27] greg-g: cool :) [19:41:29] 6Release-Engineering-Team: High increase in fatalmonitor with wmf.4 to wikipedias - https://phabricator.wikimedia.org/T117087#1766271 (10greg) 3NEW [19:42:15] 6Release-Engineering-Team: High increase in fatalmonitor with wmf.4 to wikipedias - https://phabricator.wikimedia.org/T117087#1766290 (10greg) [19:50:27] (03CR) 10Krinkle: [C: 031] Add new jsduck template [integration/config] - 10https://gerrit.wikimedia.org/r/249416 (owner: 10Paladox) [19:59:58] (03CR) 10Krinkle: "Removing myself as reviewer. I don't maintain this extension." [integration/config] - 10https://gerrit.wikimedia.org/r/249414 (owner: 10Paladox) [20:22:45] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [20:27:58] greg-g: i don't know how to help with that bug... i can't see anything obvious, can't replicate and have no knowledge of what ResourceLoader is doing here touching the database. I'm out of my depth. [20:28:11] yeah, thanks and sorry jdlrobson [20:28:19] 10Differential, 5Gerrit-Migration, 7Documentation: Create example workflows for differential showing old way and new way side by side - https://phabricator.wikimedia.org/T117058#1766483 (10Qgil) [20:28:43] greg-g: i'm sorry.. i really want to help fix it! [20:29:52] jdlrobson: :) I know, you usually do :) [21:04:16] 6Release-Engineering-Team, 3Scap3, 10Wikimedia-Developer-Summit-2016: Scap3: updates, upgrades, and challenges - https://phabricator.wikimedia.org/T114045#1766706 (10thcipriani) The first deploy via Scap3 happened on beta cluster last Wednesday (outcomes outlined here: https://www.mediawiki.org/wiki/Deployme... [21:04:56] 10Beta-Cluster-Infrastructure, 5Patch-For-Review, 5WMF-deploy-2015-11-03_(1.27.0-wmf.5): Make Privacy Policy/ToS on Beta Cluster link to the labs version (not production version) - https://phabricator.wikimedia.org/T77858#1766713 (10Krenair) a:5Krenair>3None So it seems that didn't actually change the fo... [21:05:06] 10Beta-Cluster-Infrastructure, 5WMF-deploy-2015-11-03_(1.27.0-wmf.5): Make Privacy Policy/ToS on Beta Cluster link to the labs version (not production version) - https://phabricator.wikimedia.org/T77858#1766720 (10Krenair) [21:22:14] (03PS3) 10Hashar: Add new jsduck template [integration/config] - 10https://gerrit.wikimedia.org/r/249416 (owner: 10Paladox) [21:24:22] (03CR) 10Hashar: [C: 032] "The diff in integration-zuul-layoutdiff is due to a different ordering of the jobs that comes from templates. Beside that it is a noop." [integration/config] - 10https://gerrit.wikimedia.org/r/249416 (owner: 10Paladox) [21:24:50] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/249416 (owner: 10Paladox) [21:25:30] (03Merged) 10jenkins-bot: Add new jsduck template [integration/config] - 10https://gerrit.wikimedia.org/r/249416 (owner: 10Paladox) [21:29:57] (03CR) 10Hashar: "Lets wait a bit for reviewers of source change https://gerrit.wikimedia.org/r/#/c/249410/ to acknowledge it." [integration/config] - 10https://gerrit.wikimedia.org/r/249414 (owner: 10Paladox) [21:32:51] (03PS2) 10Hashar: [MultimediaViewer] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:33:22] (03CR) 10Hashar: "Rebased to drop jsduck (it is now a template)" [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:33:39] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:37:19] (03CR) 10Hashar: "Looks fine, gotta verify whether MultimediaViewer pass the core structure tests." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:39:10] (03PS1) 10Paladox: Updates tests for extension-qunit-generic [integration/config] - 10https://gerrit.wikimedia.org/r/249891 [21:40:06] Hashar: https://gerrit.wikimedia.org/r/#/c/249891/ this patch should help us test extension unnit tests on MultimediaViewe extension. [21:40:46] paladox: yeah [21:40:58] though that apply all those jobs to any extension having extension-qunit-generic :D [21:41:31] yes but it is in experimental: so should help us for any other extension that dosent yet use extension unnit tests but uses the generic qunit test. [21:42:41] (03CR) 10Hashar: [C: 032] "Pass :)" [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:43:12] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:43:57] paladox: I have tested it on my local machine :-} [21:44:08] hashar: Ok thanks. [21:46:56] 10Beta-Cluster-Infrastructure, 5WMF-deploy-2015-11-03_(1.27.0-wmf.5): Make Privacy Policy/ToS on Beta Cluster link to the labs version (not production version) - https://phabricator.wikimedia.org/T77858#1766822 (10Krenair) Okay, turns out I was looking at a cached copy. We still need to fix the extra "By using... [21:48:27] (03Merged) 10jenkins-bot: [MultimediaViewer] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:53:21] 10Continuous-Integration-Config, 10Fundraising-Backlog: Forced merges are breaking Zuul - https://phabricator.wikimedia.org/T117062#1766875 (10hashar) Yup force merge is not supported by Zuul and that has the side effect of dead locking it. The workflow is: ``` - name: wikimedia/fundraising/crm check:... [21:54:03] 10Continuous-Integration-Config, 10Fundraising-Backlog: Tests on deployment branches of wikimedia/fundraising/crm falling causing to force merge (and deadlock of Zuul) - https://phabricator.wikimedia.org/T117062#1766887 (10hashar) [21:58:07] (03CR) 10Hashar: "Pass tests https://gerrit.wikimedia.org/r/#/c/249897/ :)" [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:58:59] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/249763 (owner: 10Paladox) [21:59:48] paladox: you are welcome :-) [22:00:00] hashar: :) [22:01:30] hashar: Would this https://gerrit.wikimedia.org/r/#/c/247920/ work. and https://gerrit.wikimedia.org/r/#/c/245495/ [22:02:27] would be for tomorrow [22:02:28] :-D [22:02:52] paladox: I looked at them, they looks fine at first glance [22:03:04] hashar: Ok Thanks. [22:03:07] paladox: I will probably just merge them [22:03:14] or amend if needed and merge [22:03:16] hashar: Thanks :) [22:03:46] paladox: if you have some writing skills, you can reach out to other developers on the wikitech-l mailing list to advertise composer/npm [22:04:05] paladox: and potentially review https://www.mediawiki.org/wiki/Continuous_integration/Entry_points :-} [22:04:20] hashar: Ok. [22:08:48] have a good evening/night/breakfast/whatever [22:30:34] (03PS1) 10Legoktm: doc: Add base_convert [integration/docroot] - 10https://gerrit.wikimedia.org/r/249909 [22:31:04] (03CR) 10Legoktm: [C: 032] doc: Add base_convert [integration/docroot] - 10https://gerrit.wikimedia.org/r/249909 (owner: 10Legoktm) [22:31:25] (03Merged) 10jenkins-bot: doc: Add base_convert [integration/docroot] - 10https://gerrit.wikimedia.org/r/249909 (owner: 10Legoktm) [23:15:44] greg-g: still a mystery? [23:16:17] jdlrobson: honestly, I stepped away from the computer for a bit and just got back a little while ago, twentyafterfour can answer more aptly [23:16:26] see also: my pissiness earlier :) [23:16:42] :( [23:17:29] what's the question? [23:17:44] oh the deadlock errors? [23:18:03] ori says it's somewhat normal, although I've never seen it before myself [23:18:19] has it mostly gone away, as he and aaron thought? [23:18:26] * greg-g could open logstash, but... [23:19:03] seemingly [23:23:54] Okay. Well let me know if you need anything specific from me! I'm on standby :) [23:32:48] 3Scap3: Include architectural overview in scap3 documentation - https://phabricator.wikimedia.org/T117131#1767304 (10dduvall) 3NEW a:3dduvall [23:33:08] 10Deployment-Systems, 3Scap3: Include architectural overview in scap3 documentation - https://phabricator.wikimedia.org/T117131#1767312 (10dduvall) [23:33:27] thanks jdlrobson [23:33:49] 10Beta-Cluster-Infrastructure, 10MediaWiki-DjVu: DjVu rendering broken on Beta - https://phabricator.wikimedia.org/T117132#1767315 (10Rillke) 3NEW [23:36:34] 10Deployment-Systems, 3Scap3: Include architectural overview in scap3 documentation - https://phabricator.wikimedia.org/T117131#1767326 (10dduvall) [23:39:37] jdlrobson: I think everything is fine, thanks!