[00:05:01] twentyafterfour is that centralauth? [00:05:04] looks like [00:05:05] it [00:05:10] ive filled a task for that [00:05:30] twentyafterfour: https://phabricator.wikimedia.org/T168802 [00:19:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 10MediaWiki-extensions-CentralAuth: CentralAuth tests are broken - https://phabricator.wikimedia.org/T168802#3384857 (10Paladox) This is blocking LoginNotify extension too due to CentralAuth error. [00:44:00] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 10MediaWiki-extensions-CentralAuth: CentralAuth tests are broken - https://phabricator.wikimedia.org/T168802#3377138 (10MaxSem) So I tracked it down to broken `tableExists()` in core, submitted https://gerrit.wi... [02:11:43] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3384928 (10Dzahn) "releases: Duplicate most of the microsite setup on new host releases1001" (Chad) - https://ger... [02:43:32] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3384973 (10Dzahn) "Swap reprepro upload role to profile" (Chad) - https://gerrit.wikimedia.org/r/#/c/361805/ [02:57:16] 10Deployment-Systems, 10MediaWiki-Internationalization, 10Performance-Team: Experiment with plain .php files for l10n cache instead of CDB - https://phabricator.wikimedia.org/T99740#3384997 (10Krinkle) [08:09:23] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:43:32] 10Gerrit, 10MediaWiki-Vagrant, 10Patch-For-Review: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#3385364 (10Tgr) >>! In T152801#3375249, @Tgr wrote: > Unfortunately it does not seem like raising the post buffer solved this, although the error became... [08:49:23] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [09:04:40] hi, what's the procedure to backport a bugfix to 1.27,1.28 & 1.29? [09:05:49] dcausse: create cherry picks to the REL1_27/8/9 branches and add Chad as a reviewer [09:06:10] legoktm: ok, thanks! [09:07:59] dcausse: I'd also recommend setting up 1.27 locally to test the cherry-picks just to make sure there are no fatals and stuff [09:08:31] legoktm: sure will do [09:40:22] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:43:30] (03PS1) 10Hashar: castor: use JJB include statements [integration/config] - 10https://gerrit.wikimedia.org/r/361839 [09:46:47] (03CR) 10Hashar: "We talked about that a while ago, this change is merely for demo purpose and is a NOOP." [integration/config] - 10https://gerrit.wikimedia.org/r/361839 (owner: 10Hashar) [10:02:03] (03PS1) 10Hashar: castor: get rid of intermediate env file [integration/config] - 10https://gerrit.wikimedia.org/r/361843 [10:06:10] !log Unblocked beta cluster jenkins job. Have been stalled for a while [10:06:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:20:23] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [11:39:22] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3385911 (10MarcoAurelio) Anything I have to do on my part or just keep waiting? :) Regards. [11:41:36] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3385935 (10Paladox) @demon could you add the missing entry please? :) https://groups.google.com/forum/#!topic/repo-discuss/1ZYXalK8iG4 [11:54:39] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Project-Admins, 10Stewards-and-global-tools, 10WMF-Legal: Create acl*stewards - https://phabricator.wikimedia.org/T131766#3386027 (10MarcoAurelio) 05Open>03Resolved a:05Jalexander>03mmodell @mmodell Thanks! I've added my fellows over there. I... [11:55:44] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Project-Admins, 10Stewards-and-global-tools, 10WMF-Legal: Create acl*stewards - https://phabricator.wikimedia.org/T131766#3386036 (10MarcoAurelio) [12:33:42] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Project-Admins, 10Stewards-and-global-tools, 10WMF-Legal: Create acl*stewards - https://phabricator.wikimedia.org/T131766#3386225 (10Aklapper) a:05Jalexander>03mmodell So leave the tag. :) [12:46:44] Project beta-code-update-eqiad build #161697: 15ABORTED in 25 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/161697/ [12:48:49] (03PS1) 10Hashar: beta-code-update-eqiad: remove global envs [integration/config] - 10https://gerrit.wikimedia.org/r/361854 [12:53:43] (03PS2) 10Hashar: beta-code-update-eqiad: remove global envs [integration/config] - 10https://gerrit.wikimedia.org/r/361854 [13:07:11] (03PS1) 10Hashar: Remove global-setup from doxygen jobs [integration/config] - 10https://gerrit.wikimedia.org/r/361856 [13:39:28] 10Beta-Cluster-Infrastructure, 10Wikimedia-Extension-setup, 10Category, 10FileAnnotations (Beta Cluster Release): Release FileAnnotations on the Beta Cluster - https://phabricator.wikimedia.org/T144302#3386487 (10Bawolff) [13:49:13] (03CR) 10Hashar: [C: 032] "Tested on oojs/ui change https://gerrit.wikimedia.org/r/#/c/361091/2" [integration/config] - 10https://gerrit.wikimedia.org/r/361856 (owner: 10Hashar) [13:54:37] (03CR) 10jerkins-bot: [V: 04-1] Remove global-setup from doxygen jobs [integration/config] - 10https://gerrit.wikimedia.org/r/361856 (owner: 10Hashar) [13:54:57] grblbllb [13:55:47] (03CR) 10Hashar: [C: 032] Remove global-setup from doxygen jobs [integration/config] - 10https://gerrit.wikimedia.org/r/361856 (owner: 10Hashar) [14:00:30] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:03:45] (03CR) 10jerkins-bot: [V: 04-1] Remove global-setup from doxygen jobs [integration/config] - 10https://gerrit.wikimedia.org/r/361856 (owner: 10Hashar) [14:03:58] !log pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python - See https://status.python.org for updates [14:04:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:05:57] 10Continuous-Integration-Infrastructure: pypi.python.org having transient issues (503 errors) - https://phabricator.wikimedia.org/T169091#3386757 (10hashar) [14:06:06] 10Continuous-Integration-Infrastructure, 10Upstream: pypi.python.org having transient issues (503 errors) - https://phabricator.wikimedia.org/T169091#3386769 (10hashar) [14:06:16] !log pypi.python.org has an issue with its CDN . That would affect any CI jobs relying on tox/python - See https://status.python.org for updates and T169091 [14:06:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:06:19] T169091: pypi.python.org having transient issues (503 errors) - https://phabricator.wikimedia.org/T169091 [14:09:57] 10Continuous-Integration-Infrastructure, 10Upstream: pypi.python.org having transient issues (503 errors) - https://phabricator.wikimedia.org/T169091#3386786 (10hashar) ``` $ dig +short ANY pypi.python.org prod.python.map.fastly.net. ``` @Paladox mentioned a similar error with apt-get: ``` lang=irc [16:06:30]... [14:10:24] 10Continuous-Integration-Infrastructure, 10Upstream: pypi.python.org having transient issues (503 errors) - https://phabricator.wikimedia.org/T169091#3386788 (10Paladox) Apparently someone broke redirects. so /debian is now /debian/ [14:10:31] 10Continuous-Integration-Infrastructure, 10Upstream: Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) ) - https://phabricator.wikimedia.org/T169091#3386791 (10hashar) [14:11:18] (03CR) 10Hashar: [C: 032] Remove global-setup from doxygen jobs [integration/config] - 10https://gerrit.wikimedia.org/r/361856 (owner: 10Hashar) [14:14:16] (03Merged) 10jenkins-bot: Remove global-setup from doxygen jobs [integration/config] - 10https://gerrit.wikimedia.org/r/361856 (owner: 10Hashar) [14:14:42] 10Continuous-Integration-Infrastructure, 10Upstream: Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) ) - https://phabricator.wikimedia.org/T169091#3386812 (10hashar) [14:33:47] !log running alter tables on the EL database in deployment-eventlogging03.deployment-prep.eqiad.wmflabs [14:33:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:35:28] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [14:42:39] 10Continuous-Integration-Infrastructure, 10Upstream: Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) ) - https://phabricator.wikimedia.org/T169091#3386962 (10hashar) 05Open>03Resolved **FIXED** https://status.fastly.com/ Jun 28, 14:40 UTC Resolved - This inc... [14:42:59] !log pypi.python.org is back again - T169091 [14:43:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:43:13] T169091: Fastly CDN has some transient issue (was: pypi.python.org having transient issues (503 errors) ) - https://phabricator.wikimedia.org/T169091 [15:02:31] 10Gerrit, 10MediaWiki-Vagrant, 10Patch-For-Review: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#3387066 (10demon) >>! In T152801#3385364, @Tgr wrote: >>>! In T152801#3375249, @Tgr wrote: >> Unfortunately it does not seem like raising the post buffe... [15:04:05] PROBLEM - Host deployment-ms-be-test01 is DOWN: CRITICAL - Host Unreachable (10.68.21.190) [15:09:12] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:26:12] (03PS1) 10Hashar: beta-update-databases: invoke global-teardown directly [integration/config] - 10https://gerrit.wikimedia.org/r/361883 (https://phabricator.wikimedia.org/T97257) [15:27:15] (03PS2) 10Hashar: beta-update-databases: invoke global-teardown directly [integration/config] - 10https://gerrit.wikimedia.org/r/361883 (https://phabricator.wikimedia.org/T97257) [15:29:05] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<55.56%) [15:33:06] !log running git gc under /srv/mediawiki-staging [15:33:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:37:08] Project beta-scap-eqiad build #161633: 04FAILURE in 3 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/161633/ [15:39:34] 10Release-Engineering-Team (Kanban), 10releng-201617-q4, 10MediaWiki-General-or-Unknown, 10MW-1.29-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T153271#3387280 (10demon) p:05Normal>03High [15:40:27] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Project-Admins, 10Stewards-and-global-tools, 10WMF-Legal: Create acl*stewards - https://phabricator.wikimedia.org/T131766#3387281 (10mmodell) The #releng-kanban tag is used to track what the release engineering team has been working on, please leave t... [15:43:58] I broke beta-scap-eqiad [15:46:48] Yippee, build fixed! [15:46:48] Project beta-scap-eqiad build #161634: 09FIXED in 3 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/161634/ [15:47:34] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Project-Admins, 10Stewards-and-global-tools, 10WMF-Legal: Create acl*stewards - https://phabricator.wikimedia.org/T131766#3387305 (10MarcoAurelio) Okay! :) [15:47:39] !log beta: git -C /srv/deployment/ores/deploy/submodules/editquality gc (saving 380MBytes) [15:47:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:52:11] (03CR) 10Hashar: [C: 032] "Tested on https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/18071/console" [integration/config] - 10https://gerrit.wikimedia.org/r/361883 (https://phabricator.wikimedia.org/T97257) (owner: 10Hashar) [15:54:46] (03Merged) 10jenkins-bot: beta-update-databases: invoke global-teardown directly [integration/config] - 10https://gerrit.wikimedia.org/r/361883 (https://phabricator.wikimedia.org/T97257) (owner: 10Hashar) [15:55:06] !log beta: git gc mediawiki repos in /srv/mediawiki-staging [15:55:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:08:51] 10Release-Engineering-Team, 10Page-Previews, 10Reading-Web-Backlog, 10Reading-Web-Kanban-Board: Create bot that automatically rebases and rebuilds patches to master - https://phabricator.wikimedia.org/T167181#3387409 (10Jhernandez) Will signoff soon. [16:13:12] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:20:33] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:28:18] RainbowSprinkles this https://gerrit-review.googlesource.com/#/c/111551/ change looks like it will make it easier to disable wmf staff accounts when they leave wmf :) [16:28:21] (03PS1) 10Hashar: Drop global-setup/global-teardown macros [integration/config] - 10https://gerrit.wikimedia.org/r/361897 [16:29:04] paladox: We don't disable ldap accounts typically :) [16:29:08] (03CR) 10Hashar: [C: 032] Drop global-setup/global-teardown macros [integration/config] - 10https://gerrit.wikimedia.org/r/361897 (owner: 10Hashar) [16:29:08] But yes, that is nice [16:29:10] for when we do [16:29:16] Oh, :) [16:30:46] (03Merged) 10jenkins-bot: Drop global-setup/global-teardown macros [integration/config] - 10https://gerrit.wikimedia.org/r/361897 (owner: 10Hashar) [16:30:52] moaaar cleanup [16:37:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (Watching / External), 10Discovery, 10Discovery-Analysis (Current work): Add lint/CI to all wikimedia/discovery analytics repositories - https://phabricator.wikimedia.org/T153856#3387558 (10hashar) Thank you @mpopov ! I am going to do the do the p... [18:13:31] 10Gerrit, 10Upstream: Change Gerrit's default indentation to tabs - https://phabricator.wikimedia.org/T168590#3387888 (10Paladox) This won't be the default, but we can add support for it in the preference. Done here https://gerrit-review.googlesource.com/#/c/111630/ [19:06:22] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Easy, 10Google-Code-In-2016, and 5 others: Use of &$this in hooks triggers warnings in PHP 7.1 - https://phabricator.wikimedia.org/T153505#3388092 (10Umherirrender) Would be nice to have a sniff for this to avoid adding new cases and to change exte... [19:10:36] 10Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 10MW-1.30-release, 10Release: Release MediaWiki 1.29 - https://phabricator.wikimedia.org/T169127#3388107 (10Luke081515) [19:10:45] 10Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 10MW-1.30-release, 10Release: Release MediaWiki 1.30 - https://phabricator.wikimedia.org/T169127#3388107 (10Luke081515) [19:13:07] 10Beta-Cluster-Infrastructure: Beta thumbnails are broken - https://phabricator.wikimedia.org/T169114#3388126 (10Luke081515) [19:18:07] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3388157 (10mmodell) [19:20:48] still getting test failures due to database table errors [19:21:02] Table 'unittest_globalnames' already exists [19:22:00] yep [19:22:15] twentyafterfour it's a centralauth problem due to a recent change in mw core [19:27:30] twentyafterfour: when you have some time, can you take a look at phab? When I'm trying to edit https://phabricator.wikimedia.org/T167893 I get "You do not have access to any forms which are enabled and marked as edit forms." [19:28:22] @sagan: that's because T107893 is a special task - the description is generated, it cannot be edited [19:28:23] T107893: Analyze webrequest data issue on August 3/4 and 10/11 [?pts] {hawk} - https://phabricator.wikimedia.org/T107893 [19:29:23] twentyafterfour: 8guess you linked the wrong task now). But I can't comment it either [19:29:41] it's created from custom code in https://phabricator.wikimedia.org/source/phab-extensions/browse/wmf%252Fstable/src/customfields/ [19:30:06] sorry yeah T167893 [19:30:07] T167893: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893 [19:30:17] SAGAN: you can't add a comment? [19:30:25] that's odd [19:30:33] wait, I will take a screenshot [19:31:58] twentyafterfour: here, that's what I can see: https://phabricator.wikimedia.org/F8548975 [19:32:13] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3388233 (10mmodell) [19:33:34] no box to comment [19:36:46] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3388275 (10mmodell) [19:37:33] SAGAN: thanks, I'll look into it, I'm sure it's just a simple misconfiguration [19:37:39] ok, thanks :) [19:38:02] twentyafterfour: the task looks a bit like it is into the status "locked" for me [19:38:37] 10Beta-Cluster-Infrastructure, 10Performance-Team, 10Availability (Multiple-active-datacenters): Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#3388293 (10aaron) >>! In T151466#3299491, @elukey wrote: > Couple of questions: > > 1) What is the scope that mcrouter will... [19:43:39] twentyafterfour: https://gerrit.wikimedia.org/r/#/c/361938/ there you go. [19:54:17] thanks James_F [19:54:49] twentyafterfour: Always happy to help. :-) [20:06:58] SAGAN: can you comment now? [20:08:10] twentyafterfour: yup, that works for me :) [20:08:25] and I can edit some things as well [20:08:56] twentyafterfour: I guess you realized that with the subtypes upstream created? [20:09:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3388376 (10mmodell) [20:09:49] SAGAN: yeah it's complicated to configure but it provides some nice new functionality [20:10:06] I just had to make an edit form for release tasks [20:10:20] previously I had a 'create release' form but no 'edit release' form [20:11:31] twentyafterfour: now there is only one small thing that is not yet perfect [20:11:53] I can't edit the description (not a problem), but the preview for it loads and loads and does not finish loading [20:17:06] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3388394 (10mmodell) [20:17:08] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10Patch-For-Review, 10User-Ryasmeen, 10Wikimedia-log-errors: Argument 2 passed to VisualEditorHooks::onDiffViewHeader() must be an instance of Revision, null given - https://phabricator.wikimedia.org/T169132#3388392 (10mmodell) 05Open>03Resolved a:... [20:19:05] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [20:21:47] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3388427 (10mmodell) [20:22:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3336016 (10mmodell) [20:22:06] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T167535#3388432 (10mmodell) [20:22:09] 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10MediaWiki-extensions-LoginNotify, 10Notifications, and 3 others: Fatal error: Class undefined: LoginNotifyPresentationModel in extensions/Echo/includes/formatters/EventPresentationModel.php... - https://phabricator.wikimedia.org/T168899#3388428 [20:22:24] 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10MediaWiki-extensions-LoginNotify, 10Notifications, and 3 others: Fatal error: Class undefined: LoginNotifyPresentationModel in extensions/Echo/includes/formatters/EventPresentationModel.php... - https://phabricator.wikimedia.org/T168899#3388434 [20:23:17] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10Patch-For-Review, 10User-Ryasmeen, 10Wikimedia-log-errors: Argument 2 passed to VisualEditorHooks::onDiffViewHeader() must be an instance of Revision, null given - https://phabricator.wikimedia.org/T169132#3388435 (10mmodell) a:05mmodell>03Jdforre... [20:37:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scoring-platform-team: deployment-sca03 can't call puppetmaster - https://phabricator.wikimedia.org/T143958#3388509 (10awight) [21:22:16] 10Release-Engineering-Team (Kanban), 10VisualEditor, 10MW-1.30-release-notes (WMF-deploy-2017-07-11_(1.30.0-wmf.8)), 10Patch-For-Review, and 2 others: Argument 2 passed to VisualEditorHooks::onDiffViewHeader() must be an instance of Revision, null given - https://phabricator.wikimedia.org/T169132#3388651 (1... [21:24:15] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T167536#3388656 (10mmodell) [21:24:23] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10VisualEditor, and 4 others: Argument 2 passed to VisualEditorHooks::onDiffViewHeader() must be an instance of Revision, null given - https://phabricator.wikimedia.org/T169132#3388652 (10mmodell) 05Resolved>03Open p:05Unbreak!>03Hi... [21:25:56] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10VisualEditor, and 4 others: Argument 2 passed to VisualEditorHooks::onDiffViewHeader() must be an instance of Revision, null given - https://phabricator.wikimedia.org/T169132#3388659 (10mmodell) A better fix as suggested by @Matmarex: V... [21:26:45] 10Release-Engineering-Team (Kanban), 10Revision-Slider, 10TCB-Team, 10VisualEditor, and 4 others: Argument 2 passed to VisualEditorHooks::onDiffViewHeader() must be an instance of Revision, null given - https://phabricator.wikimedia.org/T169132#3388661 (10mmodell) a:05Jdforrester-WMF>03mmodell [21:26:47] 10Scap: Scap3 considers service restarts to be deploys - https://phabricator.wikimedia.org/T169141#3388663 (10mobrovac) [21:27:46] thcipriani: fun one ^ [21:28:11] oh boy [21:29:57] 10Release-Engineering-Team (Kanban), 10Labs, 10Labs-Infrastructure, 10Patch-For-Review: labnet-users group can no more access labnet1001 / labnet1002 - https://phabricator.wikimedia.org/T169018#3388688 (10hashar) @Andrew I have regained access on both labnet1001 and labnet1002. Thank you! [21:30:03] 10Release-Engineering-Team (Kanban), 10Labs, 10Labs-Infrastructure, 10Patch-For-Review: labnet-users group can no more access labnet1001 / labnet1002 - https://phabricator.wikimedia.org/T169018#3388690 (10hashar) 05Open>03Resolved [21:30:47] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: Custom fields for task subtypes fatal on pages > 1 with: Query "ManiphestTaskQuery" failed to return a value from getPagingValueMap() - https://phabricator.wikimedia.org/T168556#3388691 (10mmodell) [21:31:12] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: Custom fields for task subtypes fatal on pages > 1 with: Query "ManiphestTaskQuery" failed to return a value from getPagingValueMap() - https://phabricator.wikimedia.org/T168556#3368238 (10mmodell) 05Open>03stalled [21:34:07] mobrovac: wowza, hard to think about. ok, let's see: so because a service restart makes a new tag if there is new code on tin (regardless if it is deployed to targets), if you then try to run a config-deploy, it will render the config in the wrong place -- sound right? [21:37:02] thcipriani: if deploy-local runs it, for sure [21:37:11] but essentially, yes [21:38:20] got it. OK, so from the sounds of it we need to ensure we are going to run "promote" before making an update to DEPLOY_HEAD, something along those lines... [21:38:22] * thcipriani thinks [21:38:28] thcipriani: the crux of the problem is that DEPLOY_HEAD gets updated, but nothing else [21:40:13] thcipriani: uh more info coming shorty, wait a sec as it seems now we can't deploy it at all [21:40:32] wat [21:42:03] mobrovac: grr fatal: destination path '/srv/deployment/eventstreams/deploy-cache/revs/05bcc8f6c26a33f8a7336b64a4f2ce0898617320' already exists and is not an empty directory [21:42:13] yup, dat [21:42:47] thcipriani: ok, it's because of the config refresh triggered by puppet [21:42:52] yeah [21:43:13] stashes the config file inside .git/config-files [21:43:17] in the revs dir [21:43:19] yup [21:43:46] thcipriani: i'll update the task with this [21:44:11] ok [21:45:06] thcipriani: btw, next tuesday is a holiday, so the dpeloyment pipeline meeting should probably be cancelled/moved [21:45:33] ah, right, 4th of July. [21:45:37] * thcipriani does [21:46:12] {{done}} [21:47:02] thnx [21:47:19] thanks for the reminder. [21:48:16] thcipriani: we need to get the new config for eventstreams out, so i will just create a dummy commit to its dpeloy repo to unblock myself [21:48:18] is that ok with you? [21:48:27] do you have all the info needed for this bug? [21:48:48] yes, please do [21:49:03] I can recreate this with the info you've provided [21:49:48] thank you for the analysis in the task description: detail is very helpful. [21:51:08] 10Scap: Scap3 considers service restarts to be deploys - https://phabricator.wikimedia.org/T169141#3388884 (10mobrovac) [21:51:17] kk cool [21:53:02] Anyone know about: [21:53:04] Error: 1146 Table 'jenkins_u0_mw.unittest_globaluser' doesn't exist (127.0.0.1:3306) [21:53:05] on Jenkins? [21:54:02] T168802 [21:54:02] T168802: CentralAuth tests are broken - https://phabricator.wikimedia.org/T168802 [21:56:20] yes [21:56:29] matt_flaschen it's a centralauth error [21:56:33] twentyafterfour got it [21:56:47] it's a regression in mw core. [21:58:28] paladox, what's the latest/where is that patch? T168802 only mentions a patch from Aaron, which is already merged. [21:59:01] matt_flaschen it's still broken aaron patch only makes the error different [21:59:11] that's from MaxSem [21:59:37] matt_flaschen, bashing my head against this atm [22:00:38] Project selenium-PageTriage ยป chrome,beta,Linux,BrowserTests build #436: 04FAILURE in 2 min 38 sec: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/436/ [22:01:15] Thanks, MaxSem [22:01:36] matt_flaschen, honestly, CA is such a contraption of billion hooks that I'd rather remove it from the all extensions suite [22:02:45] 10Scap, 10Patch-For-Review, 10Services (watching): Scap3 considers service restarts to be deploys - https://phabricator.wikimedia.org/T169141#3388959 (10Pchelolo) [22:03:30] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10Services (watching): Scap3 considers service restarts to be deploys - https://phabricator.wikimedia.org/T169141#3388963 (10thcipriani) p:05Triage>03High I think the tl;dr is that scap tasks run from tin that do not run the promote stage... [22:32:44] 10Release-Engineering-Team (Kanban), 10Scap, 10Services (watching): Scap3 considers service restarts to be deploys - https://phabricator.wikimedia.org/T169141#3389048 (10mobrovac) [22:52:25] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [22:53:01] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:01:33] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review: mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - https://phabricator.wikimedia.org/T67591#3389104 (10bd808) [23:43:27] 10Gerrit, 10Upstream: Change Gerrit's default indentation to tabs - https://phabricator.wikimedia.org/T168590#3389189 (10Huji) Equally good, if not better! [23:48:46] (03PS1) 10Thcipriani: operations-puppet-docker: use castor cache [integration/config] - 10https://gerrit.wikimedia.org/r/362106 [23:57:34] (03CR) 10Thcipriani: [C: 032] operations-puppet-docker: use castor cache [integration/config] - 10https://gerrit.wikimedia.org/r/362106 (owner: 10Thcipriani) [23:58:23] 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10Wikimedia-log-errors: Invalid parameter for message "pageinfo-subpages-value": a:1:{s:3:"num";N;} - https://phabricator.wikimedia.org/T92714#3389229 (10demon) 05Open>03Resolved a:03demon Assuming this was fixed. [23:59:08] (03Merged) 10jenkins-bot: operations-puppet-docker: use castor cache [integration/config] - 10https://gerrit.wikimedia.org/r/362106 (owner: 10Thcipriani)