[00:01:24] 10MediaWiki-Codesniffer, 10MediaWiki-General, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10MW-1.35-release, and 2 others: Fix all functions that have required parameters following optional ones - https://phabricator.wikimedia.org/T248078 (10Reedy) 05Open→03Resolved a:03Reedy Marking resolved [00:02:41] 10Beta-Cluster-Infrastructure, 10Puppet: puppetdb on deployment-puppetdb03 keeps getting OOMKilled - https://phabricator.wikimedia.org/T248041 (10Mholloway) And again today. [00:51:47] legoktm: the installer dies for me on the mw-config/?page=Options page [00:51:50] and I can't figure out why [00:51:54] HTTP 500, nothing in the logs [00:52:01] on master? [00:52:02] last lineo of output:
  • half way a tag [00:52:17] yes, master [00:52:39] HEAD is at be7af6e54e [00:53:35] works for me [00:53:37] maybe it's OOMing? [00:54:12] I've added error_reporting(E_ALL); ini_set( 'display_errors', 1 ); to mw-config/index.php [00:54:16] php 7.2, mediawiki-docker-dev [00:54:21] 8GB of ram to the container [00:54:27] but still nothing [00:54:40] always dies at the same character. [00:55:40] 17:52:02 last lineo of output:
  • I'm using ./maintenance/dev/start.sh with 7.4 [00:56:20] https://phabricator.wikimedia.org/P12099 [00:56:32] it dies a few lines later when I modify the code in an insignificant way [00:56:52] actually, no same line [00:57:01] a few characters earlier [00:57:09] so I guess memory related indeed [00:57:12] weird thoguh, never had this before [00:58:00] the line is supposed to be
  • Complete!
    • [00:58:11] yeah [00:58:24] can you use activity monitor to see if it's an actual OOM? [00:58:28] 10Beta-Cluster-Infrastructure, 10Documentation: Clarify where to find error logs in beta cluster - https://phabricator.wikimedia.org/T259081 (10Reedy) [00:59:02] do you have a lot of extensions/skins checked out? because that page will read all of them which is usually slow [01:01:17] legoktm: hmn.. maybve that's it. [01:02:34] it seemms weird that it would OOM with ~ 30 extensions [01:05:00] I have everything in Gerrit checked out, it's slow, but no issues [01:10:08] works fine now after having torn down and re-creating my dev environment [01:10:11] *shrug* [01:10:31] btw, there is definitely a styling issue still. the line-height hasn't been the same since we stopped loading Vector styles [01:10:36] font size is off as well [01:10:57] too large font size and too narrow line height [01:11:43] ugh, I see we still haven't fixed the redirect bug either [01:11:54] at the end of the install I'm automatically redirected away from the "Download LS.php" page [01:11:58] to a dead end [01:11:59] http://default.web.mw.localhost/mediawiki/mw-config/?localsettings=1 [01:12:03] (note the missing :8080) [01:12:40] anyway, ignoring for now to focus on the other three bugs I was working on [01:24:25] btw, the above was to test https://gerrit.wikimedia.org/r/575293 [01:24:31] which is now landing, another 1.35 blocker [01:32:37] (03CR) 10Krinkle: [C: 03+2] jjb: Add 'assert-no-mediawiki-errors' to quibble jobs (take 2) [integration/config] - 10https://gerrit.wikimedia.org/r/616923 (https://phabricator.wikimedia.org/T246358) (owner: 10Krinkle) [01:33:45] (03Merged) 10jenkins-bot: jjb: Add 'assert-no-mediawiki-errors' to quibble jobs (take 2) [integration/config] - 10https://gerrit.wikimedia.org/r/616923 (https://phabricator.wikimedia.org/T246358) (owner: 10Krinkle) [01:35:18] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos - https://phabricator.wikimedia.org/T246763 (10Krinkle) Still seen: James_F around? [05:36:22] (03PS1) 10Legoktm: doc: Add GlobalWatchlist [integration/docroot] - 10https://gerrit.wikimedia.org/r/616990 [05:37:25] (03CR) 10DannyS712: doc: Add GlobalWatchlist (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/616990 (owner: 10Legoktm) [05:41:00] (03CR) 10Legoktm: doc: Add GlobalWatchlist (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/616990 (owner: 10Legoktm) [05:41:09] (03PS2) 10Legoktm: doc: Add GlobalWatchlist [integration/docroot] - 10https://gerrit.wikimedia.org/r/616990 [05:44:17] (03CR) 10Legoktm: [C: 03+2] doc: Add GlobalWatchlist [integration/docroot] - 10https://gerrit.wikimedia.org/r/616990 (owner: 10Legoktm) [05:44:56] (03Merged) 10jenkins-bot: doc: Add GlobalWatchlist [integration/docroot] - 10https://gerrit.wikimedia.org/r/616990 (owner: 10Legoktm) [06:14:09] (03CR) 10Legoktm: [C: 03+2] wmf-config-wg-vars.js: Use the '/deployed' preset for Code search [tools/code-utils] - 10https://gerrit.wikimedia.org/r/616643 (https://phabricator.wikimedia.org/T248866) (owner: 10Krinkle) [06:16:29] (03CR) 10Legoktm: [C: 03+2] "> Patch Set 2:" [tools/code-utils] - 10https://gerrit.wikimedia.org/r/616643 (https://phabricator.wikimedia.org/T248866) (owner: 10Krinkle) [06:42:01] are there like two servers backing doc.wm.o now? [06:42:15] https://doc.wikimedia.org/cover-extensions/GlobalWatchlist/ shows up and then it disappears again [06:42:23] and randomly some CSS is missing [06:57:01] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [07:07:01] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [07:54:39] 10Continuous-Integration-Config, 10Research, 10Patch-For-Review: Enable CI on research/landing-page repo - https://phabricator.wikimedia.org/T257843 (10hashar) As @Reedy mentioned, committing the `package-lock.json` would make the operation potentially faster and also well ... locks the packages at a known v... [08:46:02] legoktm: Varnish caching of doc1001 is time-based without purges. [08:46:09] legoktm: So you'll get whatever the cache says. [08:47:12] how long are they cached for? Because the tests have been there fore a while [08:50:54] James_F question. at https://doc.wikimedia.org/GlobalWatchlist/master/js/#!/api/GlobalWatchlistWikibaseHandler the second method documented (counting the constructor as the first) is missing its name (`getEntityIds`) - any idea why? [08:51:29] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10thiemowmde) I'm certainly ignoring many of the very relevant arguments listed above. However, as far as I'm conc... [08:52:56] DannyS712: 24 hours, I think. [08:54:01] 10Gerrit, 10Patch-For-Review: Unbreak gerrit-replica web UI on gerrit2001 - https://phabricator.wikimedia.org/T256567 (10hashar) In another task I looked at having some bots to use `gerrit query` on the replica instead of on the master and asked upstream about it. The issue is that there is no secondary index... [08:54:01] DannyS712: Huh. No. Odd. [08:55:31] maxage=3600 so one hour [08:55:50] hashar: I think that's the frontend cache length, not the backend one? [08:55:56] * James_F shrugs. [08:56:17] https://phabricator.wikimedia.org/T252131 [08:56:18] okay, the tests at https://doc.wikimedia.org/cover-extensions/GlobalWatchlist/ have definitely been there longer than a day, and sometimes it loads them, but other times it doesn't. Didn't see this with any other extension [08:56:28] reedy noticed the issue a few weeks ago [08:56:48] Yeah [08:57:25] that page gives me an Empty directory [08:57:52] date: Wed, 29 Jul 2020 08:56:57 GMT [08:57:55] which is well .. right now [08:58:02] thats the problem - it isn't empty, it has a lot of test coverage [08:58:08] x-cache [08:58:08] cp3054 hit, cp3054 pass [08:58:08] x-cache-status [08:58:08] hit-local [08:58:31] and it is a HTTP 200 [08:59:26] and [08:59:26] Is the front-end somehow hitting doc2001 rather than 1001 or something odd? [08:59:55] I don't think we have a doc2001 [09:00:05] on the server [09:00:09] Wasn't Daniel replacing it with a buster one [09:00:14] Or is that already done now? [09:00:22] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) a:05brennen→03LarsWirzenius [09:01:24] /srv/docroot/org/wikimedia/doc/cover-extensions/GlobalWatchlist/ [09:01:32] it only hsa a single file clover.xml [09:01:41] so the HTML coverage report hasn't been generated [09:01:53] Ah. [09:02:01] there is no index.html or index.php or whatever [09:02:23] hmm. How should I be causing it to be generated? [09:02:39] is there a post-merge task I missed in the zuul config? [09:02:45] well [09:02:57] stuff got generated and published, so the zuul config is probably fine [09:03:14] but maybe something is missing in mediawiki/extensions/GlobalWatchlist [09:04:35] https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-docker-publish/26600/console [09:04:38] 00:01:22.179 Generating code coverage report in HTML format ... done [21 ms] [09:04:58] Generated but not copied? [09:05:07] Is it put in the wrong dir? [09:05:07] https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-docker-publish/26600/artifact/log/clover.xml [09:05:12] that file got copied [09:05:57] ah yeah maybe [09:05:58] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Platform Engineering, 10Quibble, and 3 others: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10Jdfo... [09:06:13] https://integration.wikimedia.org/ci/job/publish-to-doc1001/80844/console has the rsync command [09:06:29] rsync ... jenkins-deploy@172.16.7.210:/srv/jenkins/workspace/workspace/mwext-phpunit-coverage-docker-publish/cover/. . [09:10:41] hmm [09:10:44] or we broke something ehehe [09:11:15] Wikibase, MobileFrontend, MachineVision, EntitySchema all have the same issue [09:12:10] but Translate or ContentTranslation had patches merged today and have a html coverage report [09:14:07] relatively to mediawiki/core $IP , the job runs: php -d zend_extension=xdebug.so tests/phpunit/phpunit.php --testsuite extensions --coverage-clover ./log/clover.xml --coverage-html ./cover --log-junit ./log/junit.xml ./extensions/GlobalWatchlist/tests/phpunit [09:14:33] DannyS712: you can try to reproduce locally and check whether the HTML coverage report is properly generated? [09:15:29] what do I need to do to reproduce it? I use windows and if I try to `php` something it results in bash complaining [09:17:03] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Platform Engineering, 10Quibble, and 3 others: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10hash... [09:17:09] also, any ideas why `global` is one of the entries on the sidebar at https://doc.wikimedia.org/GlobalWatchlist/master/js/#!/api ? it isn't for, eg, https://doc.wikimedia.org/mediawiki-core/master/js/#!/api [09:17:10] bah ;D [09:19:11] DannyS712: lets file a task about it [09:19:16] will dig into it this afternoon [09:20:24] sure [09:20:32] do you want me to file the task, or can you? [09:20:42] can you ? [09:20:45] I gotta escape [09:20:53] but I am pretty sure I can reproduce it easily [09:20:56] rm -f src/LocalSettings.php; ZUUL_PROJECT=mediawiki/extensions/GlobalWatchlist quibble -c 'php -d zend_extension=xdebug.so tests/phpunit/phpunit.php --testsuite extensions --coverage-clover ./log/clover.xml --coverage-html ./cover --log-junit ./log/junit.xml ./extensions/GlobalWatchlist/tests/phpunit' [09:21:17] I have to meet a neighboor to grab some piece of furniture [09:21:34] and we should get your php fixed ;] [09:22:12] 10Gerrit, 10Patch-For-Review: Use WMF Gerrit's logo as favicon - https://phabricator.wikimedia.org/T257218 (10Lucas_Werkmeister_WMDE) @Tgr that looks fine by me, thanks :) [09:22:53] no idea what I should be tagging? [09:22:57] https://phabricator.wikimedia.org/T259109 [09:23:15] #continuous-integration-config would work [09:23:39] 10Continuous-Integration-Config, 10User-DannyS712: Some extensions' code coverage reports fail with `Empty directory!` - https://phabricator.wikimedia.org/T259109 (10DannyS712) [09:37:30] DannyS712: thank you, will dig into it this afterrnoon ;) [09:37:32] * hashar vanishes [10:07:46] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Noted that {T157997} seems to have happened a couple of times. [10:11:56] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Reported {T259111}, but it's only happened a couple of times so far... [10:15:56] 10Continuous-Integration-Config, 10User-DannyS712: Some extensions' code coverage reports fail with `Empty directory!` - https://phabricator.wikimedia.org/T259109 (10hashar) On doc1001 ` /srv/docroot/org/wikimedia/doc/cover-extensions/GlobalWatchlist/` only has the `clover.xml` file. Some coverage seems to ha... [10:23:38] 10Continuous-Integration-Config, 10User-DannyS712: Some extensions' code coverage reports fail with `Empty directory!` - https://phabricator.wikimedia.org/T259109 (10hashar) I ran the job again adding some debug statement: ` + find . -name cover + find . -name index.html ` And it worke... [11:29:22] 10Continuous-Integration-Config, 10User-DannyS712: Some extensions' code coverage reports fail with `Empty directory!` - https://phabricator.wikimedia.org/T259109 (10hashar) Everything looks fine as far as I can tell, at least in how the coverage is generated, directories paths and logic. The publish-to-doc10... [11:46:17] (03PS1) 10Hashar: jjb: enhance rsync output in publish-to-doc1001 [integration/config] - 10https://gerrit.wikimedia.org/r/617097 (https://phabricator.wikimedia.org/T259109) [11:48:47] (03PS2) 10Hashar: jjb: enhance rsync output in publish-to-doc1001 [integration/config] - 10https://gerrit.wikimedia.org/r/617097 (https://phabricator.wikimedia.org/T259109) [11:49:24] (03CR) 10Hashar: "Rephrased the commit message to add some more details, notably an example output for 'rsync --stats'" [integration/config] - 10https://gerrit.wikimedia.org/r/617097 (https://phabricator.wikimedia.org/T259109) (owner: 10Hashar) [11:53:02] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [12:03:02] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [12:05:40] (03CR) 10Hashar: "That does not fix the issue but will definitely helps track it down ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/617097 (https://phabricator.wikimedia.org/T259109) (owner: 10Hashar) [12:18:46] (03PS1) 10Hashar: jjb: remove legacy castor job [integration/config] - 10https://gerrit.wikimedia.org/r/617108 [12:21:59] (03CR) 10Hashar: "In a chat with Ahmon earlier this week, he enquired about that nodepool credential. Turns out the job using it is no more used anywhere ;" [integration/config] - 10https://gerrit.wikimedia.org/r/617108 (owner: 10Hashar) [12:30:17] 10Continuous-Integration-Infrastructure, 10Operations, 10Jenkins: Update Jenkins gpg release key in reprepro - https://phabricator.wikimedia.org/T259116 (10hashar) [12:33:35] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10User-DannyS712: Some extensions' code coverage reports fail with `Empty directory!` - https://phabricator.wikimedia.org/T259109 (10hashar) a:03hashar [12:35:06] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10User-DannyS712: Some extensions' code coverage reports fail... - https://phabricator.wikimedia.org/T259109 [13:02:56] (03PS2) 10Jforrester: Run node10-docker on wikimedia/security/landing-page [integration/config] - 10https://gerrit.wikimedia.org/r/612285 (https://phabricator.wikimedia.org/T257837) (owner: 10Reedy) [13:05:56] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Promoted to group1. [13:07:38] (03CR) 10Jforrester: [C: 03+2] Run node10-docker on wikimedia/security/landing-page [integration/config] - 10https://gerrit.wikimedia.org/r/612285 (https://phabricator.wikimedia.org/T257837) (owner: 10Reedy) [13:08:13] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) As it is now, 1.35 supports both 7.2 and 7.3. So the goal is to get out of the situation where code using M... [13:08:43] (03Merged) 10jenkins-bot: Run node10-docker on wikimedia/security/landing-page [integration/config] - 10https://gerrit.wikimedia.org/r/612285 (https://phabricator.wikimedia.org/T257837) (owner: 10Reedy) [13:09:03] !log Zuul: Installing CI for wikimedia/security/landing-page T257837 [13:09:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:09:05] T257837: Enable CI for security.wikimedia.org - https://phabricator.wikimedia.org/T257837 [13:15:43] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10RazeSoldier) >>! In T257879#6344561, @Tgr wrote: > As it is now, 1.35 supports both 7.2 and 7.3. Can't it be an... [13:26:26] 10Release-Engineering-Team, 10Quibble, 10Testing-Roadblocks: Modify Quibble to accept a LocalSettings.php file - https://phabricator.wikimedia.org/T235865 (10eprodromou) [13:27:14] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) >>! In T257879#6344581, @RazeSoldier wrote: > Can't it be announced that 1.35 supports 7.4? It actually do... [13:33:46] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Release 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259128 (10Reedy) [13:34:32] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Release 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259128 (10Reedy) [13:35:04] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Write release announcement for 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259129 (10Reedy) [13:36:42] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Release 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259128 (10Reedy) [13:36:44] 10MediaWiki-Releasing, 10MW-1.35-notes, 10MW-1.35-release, 10Platform Team Workboards (Initiatives): Release MW 1.35 - https://phabricator.wikimedia.org/T256375 (10Reedy) [13:36:55] 10MediaWiki-Releasing, 10MW-1.35-notes, 10MW-1.35-release, 10Platform Team Workboards (Initiatives): Release MW 1.35.0 - https://phabricator.wikimedia.org/T256375 (10Reedy) [13:39:21] Reedy: a task! [13:39:41] JFDI [13:41:01] Reedy: we follow when you release stuff. It's nice to be able to give people a task number and say this is what we know. [13:50:26] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10thiemowmde) > […] code using MB/PCRE methods need to account for two different behaviors, and might end up worki... [14:00:50] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Write release announcement for 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259129 (10Reedy) [14:02:02] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Write release announcement for 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259129 (10Reedy) [14:14:44] 10Continuous-Integration-Config, 10Security-Team: Enable CI for security.wikimedia.org - https://phabricator.wikimedia.org/T257837 (10Reedy) 05Open→03Resolved a:03Reedy Probably more changes later, but this is the bare minimum [15:05:22] 10Release-Engineering-Team, 10Analytics-Radar, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) [15:06:08] 10Release-Engineering-Team, 10Analytics-Radar, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) 05Open→03Resolved Thanks, Luca! Just ran `sudo -u analytics-product kerberos-run-command analytic... [15:34:22] 10Gerrit, 10Patch-For-Review: Use WMF Gerrit's logo as favicon - https://phabricator.wikimedia.org/T257218 (10jlinehan) >>! In T257218#6342783, @Tgr wrote: > How about putting that in the Wikimedia circle? The colors would still be pretty distinctive, but it would have some level of Wikimedia identity. It's no... [15:49:10] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Rolled back to group0 due to {T259111} occurring too much. [15:57:05] (03CR) 10Ahmon Dancy: "I made this same change in https://gerrit.wikimedia.org/r/c/integration/config/+/616594 but I also removed the unreferenced jjb/castor-sav" [integration/config] - 10https://gerrit.wikimedia.org/r/617108 (owner: 10Hashar) [16:03:18] 10Gerrit, 10Regression: Clicking on a project name in the "Repo" column, only open patches are shown for that repo, not any patches as in 2.15 - https://phabricator.wikimedia.org/T256562 (10Dvorapa) But that would just show all things (even abandoned) and not in distinct categories, which is usually quite clut... [16:21:02] 10Continuous-Integration-Infrastructure, 10Zuul, 10Patch-For-Review: Add support for ecdsa keys in zuul (Also update paramiko to 2.2+) - https://phabricator.wikimedia.org/T171165 (10Dzahn) p:05Low→03Medium Raising the priority because T240266#6289890 identified it as a blocker for T240266 which we announ... [16:22:17] 10Continuous-Integration-Infrastructure, 10Zuul, 10Patch-For-Review: Add support for ecdsa keys in zuul (Also update paramiko to 2.2+) - https://phabricator.wikimedia.org/T171165 (10Dzahn) quoting Hashar "//There is currently a 1024 RSA key, hopefully Paramiko would support a 2048 bits one.//" What we want... [16:22:45] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10MediaWiki-backport-deployments: Cancel EU BACKPORT window on Tuesdays when train is run from the EU - https://phabricator.wikimedia.org/T254745 (10LarsWirzenius) [16:22:47] 10Gerrit, 10Repository-Admins: Decide whether requests for new Gerrit repos are handled on mediawiki.org or in Phabricator (currently it's both) - https://phabricator.wikimedia.org/T259147 (10Aklapper) [16:35:46] 10Release-Engineering-Team-TODO, 10User-dancy: Running seemingly read-only aptly commands as root changes file ownership - https://phabricator.wikimedia.org/T259151 (10dancy) [16:36:39] 10Continuous-Integration-Infrastructure, 10Operations, 10serviceops: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:41:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Zuul, 10observability, and 2 others: Monitor /var/log/zuul/error.log - https://phabricator.wikimedia.org/T258821 (10dancy) a:03dancy [16:42:12] 10Continuous-Integration-Infrastructure, 10Operations, 10serviceops: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:42:42] 10Continuous-Integration-Infrastructure, 10Operations, 10serviceops: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:42:53] 10Continuous-Integration-Infrastructure, 10Operations, 10serviceops: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:44:25] 10Continuous-Integration-Infrastructure, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10serviceops: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:50:06] 10Phabricator: Spaces request for ...Tech Sr Leadership - https://phabricator.wikimedia.org/T259156 (10debt) [16:52:08] 10Continuous-Integration-Infrastructure, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10serviceops: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10hashar) The error message refers to `push-no... [16:55:31] 10Gerrit, 10Regression: Clicking on a project name in the "Repo" column, only open patches are shown for that repo, not any patches as in 2.15 - https://phabricator.wikimedia.org/T256562 (10Dvorapa) I mean this is a fair point to me. Project Dashboards are much more novice-friendly than just list of open chang... [16:56:06] 10Continuous-Integration-Infrastructure, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10serviceops: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) >>! In T259152#6345663, @hashar wrote... [17:13:20] 10Phabricator (Search), 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10User-MModell, 10User-brennen: Make search context highlights work with the ferret search ... - https://phabricator.wikimedia.org/T230787 [17:13:21] 10Phabricator (2020-01-23): No results for specific string in Global Phab Search, while Advanced Search lists expected results (due to ElasticSearch for Global Search vs Ferret for Maniphest Search)? - https://phabricator.wikimedia.org/T224082 (10mmodell) [17:13:25] 10Phabricator, 10Release-Engineering-Team (Development services), 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10mmodell) [17:16:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Zuul, 10observability, and 2 others: Monitor /var/log/zuul/error.log - https://phabricator.wikimedia.org/T258821 (10thcipriani) p:05Triage→03Medium [17:18:08] 10Phabricator, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Operations, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) So I think that this i... [17:43:47] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 715 bytes in 0.053 second response time [17:51:28] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Correction: it was {T259126} happening too much, not {T258111}. Sorry. [18:04:15] (03PS1) 10QChris: Bump gerrit.war to 3.2.3-1-g185bdc3a69 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617205 [18:04:37] 10phan, 10MediaWiki-General: Remove suppression of PhanPossiblyUndeclaredVariable issue in core - https://phabricator.wikimedia.org/T259172 (10Umherirrender) [18:05:42] (03CR) 10QChris: "The corresponding WAR has been uploaded to our archiva already." [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617205 (owner: 10QChris) [18:12:26] (03CR) 10Ahmon Dancy: [C: 03+1] "> I have lost access to my gpg key (forgot the passphrase ...). If you feel adventurous you could delete the tag I created and push a new " [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/616858 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [18:12:44] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10hashar) Responding for CI / release engineering. **TLDR: support 7.2 and drop support for it later on as it bec... [18:13:29] (03PS1) 10QChris: Bring back jsonevent-layout library [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 [18:13:51] (03CR) 10jerkins-bot: [V: 04-1] Bring back jsonevent-layout library [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 (owner: 10QChris) [18:15:48] (03CR) 10QChris: "We this change to enable json logging." [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 (owner: 10QChris) [18:16:02] (03PS1) 10Krinkle: Add CI jobs for jquery-client [integration/config] - 10https://gerrit.wikimedia.org/r/617207 [18:16:48] (03CR) 10QChris: [V: 03+1 C: 03+1] Bring back jsonevent-layout library [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 (owner: 10QChris) [18:17:09] (03CR) 10jerkins-bot: [V: 04-1] Bring back jsonevent-layout library [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 (owner: 10QChris) [18:19:14] (03CR) 10Jforrester: "Did this repo really get created as a top-level one? Eurgh." [integration/config] - 10https://gerrit.wikimedia.org/r/617207 (owner: 10Krinkle) [18:20:17] (03CR) 10QChris: "Just in case searching no longer finds the commit at some point in the future. This is" [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617205 (owner: 10QChris) [18:25:48] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) [18:27:50] Project mwcore-phpunit-coverage-master build #816: 04FAILURE in 3 hr 25 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/816/ [18:29:07] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) We reported {T157997} but it's not a blocker as it doesn't currentl... [18:30:44] maintenance-disconnect-full-disks build 198485 integration-agent-docker-1012 (/: 17%, /srv: 100%, /var/lib/docker: 52%): OFFLINE due to disk space [18:33:22] (03CR) 10Krinkle: [C: 03+2] "Aye yeah, wasn't sure where else to put it. I might move a few more small ones, I'll chat on IRC about where to put those :)" [integration/config] - 10https://gerrit.wikimedia.org/r/617207 (owner: 10Krinkle) [18:34:20] (03Merged) 10jenkins-bot: Add CI jobs for jquery-client [integration/config] - 10https://gerrit.wikimedia.org/r/617207 (owner: 10Krinkle) [18:35:31] maintenance-disconnect-full-disks build 198486 integration-agent-docker-1012 (/: 17%, /srv: 78%, /var/lib/docker: 54%): RECOVERY disk space OK [18:35:45] PROBLEM - Free space - all mounts on integration-agent-docker-1012 is CRITICAL: CRITICAL: integration.integration-agent-docker-1012.diskspace._srv.byte_percentfree (<30.00%) [18:36:51] 10Phabricator: Spaces request for ...Tech Sr Leadership - https://phabricator.wikimedia.org/T259156 (10Aklapper) As I'd like to avoid overlap, I'd first welcome clarification how S14 (requested in {T175094}) and S16 (requested in {T179616}) would be related. [18:37:16] (03CR) 10Hashar: [C: 03+2] "Turns out that even with a signed tag, pbr does not recognizes a local version scheme (the +wmf1 suffix). The tag is definitely checked bu" [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/616858 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [18:37:47] J!log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/617207 [18:37:56] (03CR) 10Hashar: [V: 03+2 C: 03+2] WMF: use gear from our forked version [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/616858 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [18:38:34] (03CR) 10Hashar: "Next step is to rebuild the wheels in integration/zuul/deploy , will do that either later this night or tomorrow morning ;)" [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/616858 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [18:39:30] (03CR) 10QChris: "The relevant bug on Phabricator is https://phabricator.wikimedia.org/T259135" [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617205 (owner: 10QChris) [18:40:30] (03CR) 10QChris: [V: 03+1 C: 03+1] "The relevant task in Phabricator is https://phabricator.wikimedia.org/T259135" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 (owner: 10QChris) [18:40:58] (03PS2) 10QChris: Bump gerrit.war to 3.2.3-1-g185bdc3a69 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617205 (https://phabricator.wikimedia.org/T259135) [18:45:43] RECOVERY - Free space - all mounts on integration-agent-docker-1012 is OK: OK: All targets OK [18:52:34] (03CR) 10Thcipriani: [C: 03+1] Bump gerrit.war to 3.2.3-1-g185bdc3a69 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617205 (https://phabricator.wikimedia.org/T259135) (owner: 10QChris) [18:57:30] (03CR) 10QChris: [V: 03+2 C: 03+2] Bump gerrit.war to 3.2.3-1-g185bdc3a69 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617205 (https://phabricator.wikimedia.org/T259135) (owner: 10QChris) [18:59:14] Just a heads up that I'll restart Gerrit shortly to deploy a security fix https://phabricator.wikimedia.org/T259135 [19:00:35] thanks [19:02:52] ACKNOWLEDGEMENT - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1529 bytes in 0.008 second response time daniel_zahn restart for config change by chris https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [19:03:24] Thank for the ACK [19:03:47] ACKNOWLEDGEMENT - SSH access on gerrit1001 is CRITICAL: connect to address 208.80.154.137 and port 29418: Connection refused daniel_zahn restarting https://wikitech.wikimedia.org/wiki/Gerrit [19:05:45] (03PS1) 10Krinkle: jjb: Skip assert-no-mediawiki-errors on REL1_31 and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/617247 (https://phabricator.wikimedia.org/T259123) [19:05:56] (03PS2) 10Krinkle: jjb: Skip assert-no-mediawiki-errors on REL1_31 and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/617247 (https://phabricator.wikimedia.org/T259123) [19:06:58] (03CR) 10jerkins-bot: [V: 04-1] jjb: Skip assert-no-mediawiki-errors on REL1_31 and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/617247 (https://phabricator.wikimedia.org/T259123) (owner: 10Krinkle) [19:07:46] (03PS3) 10Krinkle: jjb: Skip assert-no-mediawiki-errors on REL1_31 and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/617247 (https://phabricator.wikimedia.org/T259123) [19:07:55] (03PS4) 10Krinkle: jjb: Skip assert-no-mediawiki-errors on REL1_31 and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/617247 (https://phabricator.wikimedia.org/T259123) [19:11:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10cscott) [19:14:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10cscott) This is likely some code added which caused a regression in the `parserTests.php` CLI script,... [19:17:04] 10phan, 10MediaWiki-General, 10Patch-For-Review: Remove suppression of PhanPossiblyUndeclaredVariable issue in core - https://phabricator.wikimedia.org/T259172 (10Daimona) Just confirming that the issue was suppressed in order for the upgrade commit to be easier to review (and because I didn't check all the... [19:21:46] (03CR) 10Krinkle: [C: 03+2] jjb: Skip assert-no-mediawiki-errors on REL1_31 and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/617247 (https://phabricator.wikimedia.org/T259123) (owner: 10Krinkle) [19:22:20] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10Krinkle) [19:22:52] (03Merged) 10jenkins-bot: jjb: Skip assert-no-mediawiki-errors on REL1_31 and REL1_34 [integration/config] - 10https://gerrit.wikimedia.org/r/617247 (https://phabricator.wikimedia.org/T259123) (owner: 10Krinkle) [19:23:08] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10brennen) [19:25:57] 10phan, 10MediaWiki-Core-Testing, 10Patch-For-Review: Remove suppression of PhanPossiblyUndeclaredVariable issue in core - https://phabricator.wikimedia.org/T259172 (10Krinkle) [19:28:39] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Platform Engineering, 10Quibble, and 3 others: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10csco... [19:42:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10Krinkle) Most the things we encountered in core as part of T246358 were PHP warnings that bypassed PHP... [19:45:42] (03CR) 10QChris: [V: 03+2 C: 03+2] "Since it's deployed already, I'll self-merge this here." [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/617206 (owner: 10QChris) [19:47:30] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Patch-For-Review, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10cscott) https://gerrit.wikimedia.org/r/617252 has a WIP patch which ensures the... [19:49:58] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Write release announcement for 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259129 (10Legoktm) [19:56:56] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10User-DannyS712: Some extensions' code coverage reports fail... - https://phabricator.wikimedia.org/T259109 [19:58:30] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Patch-For-Review, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10cscott) Best guess at this point is that this is a regression caused by 435d5f4d... [20:00:26] RECOVERY - Host integration-cumin-01 is UP: PING OK - Packet loss = 0%, RTA = 0.77 ms [20:09:50] 10Phabricator: Spaces request for ...Tech Sr Leadership - https://phabricator.wikimedia.org/T259156 (10debt) ahh, yes. We had created those two boards several years ago and stopped using them a year or two ago. The new board / project would basically be for the same usage as those older boards. Would it be easie... [20:10:13] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Patch-For-Review, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10cscott) Ok, patch works: https://gerrit.wikimedia.org/r/617252 -- still don't k... [20:10:20] 10Release-Engineering-Team, 10Prod-Kubernetes, 10serviceops, 10Kubernetes: CI pipeline/job to build and release helm chart artifacts - https://phabricator.wikimedia.org/T257333 (10jeena) I talked with @dduvall on how we could add this to CI. I initially thought we could do this pre-merge in CI, but another... [20:45:54] PROBLEM - Host integration-cumin-01 is DOWN: CRITICAL - Host Unreachable (172.16.1.185) [20:59:55] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Parsoid, 10Patch-For-Review, 10Regression: Parsoid CI is failing when parser tests are run - https://phabricator.wikimedia.org/T259186 (10cscott) Ah -- the culprit is `ParserTestRunner::setupDatabase`, whose phpdoc com... [21:16:54] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) >>! In T257879#6346169, @hashar wrote: > Once 7.2 is end of life and if it becomes too complicated to keep... [21:20:33] maintenance-disconnect-full-disks build 198519 integration-agent-docker-1010 (/: 19%, /srv: 97%, /var/lib/docker: 59%): OFFLINE due to disk space [21:22:41] PROBLEM - Free space - all mounts on integration-agent-docker-1010 is CRITICAL: CRITICAL: integration.integration-agent-docker-1010.diskspace._srv.byte_percentfree (<50.00%) [21:25:32] maintenance-disconnect-full-disks build 198520 integration-agent-docker-1010 (/: 19%, /srv: 99%, /var/lib/docker: 62%): still OFFLINE due to disk space [21:30:46] maintenance-disconnect-full-disks build 198521 integration-agent-docker-1010 (/: 19%, /srv: 48%, /var/lib/docker: 47%): RECOVERY disk space OK [21:37:42] RECOVERY - Free space - all mounts on integration-agent-docker-1010 is OK: OK: All targets OK [21:39:13] 10MediaWiki-Releasing, 10MW-1.35-release, 10Release: Write release announcement for 1.35.0-rc.0 - https://phabricator.wikimedia.org/T259129 (10Reedy) [21:40:51] RECOVERY - Host integration-cumin-01 is UP: PING OK - Packet loss = 0%, RTA = 2.05 ms [21:45:52] PROBLEM - Host integration-cumin-01 is DOWN: CRITICAL - Host Unreachable (172.16.1.185) [22:00:56] (03CR) 10Jeena Huneidi: Convert kask ci to use pipelinelib (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/616177 (https://phabricator.wikimedia.org/T224041) (owner: 10Jeena Huneidi) [22:01:26] (03PS2) 10Jeena Huneidi: Convert kask ci to use pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/616177 (https://phabricator.wikimedia.org/T224041) [22:15:50] RECOVERY - Host integration-cumin-01 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [22:20:39] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10brennen) We've reached today's North America cutoff for deploys. Assuming a resol... [22:20:51] PROBLEM - Host integration-cumin-01 is DOWN: CRITICAL - Host Unreachable (172.16.1.185) [22:55:51] RECOVERY - Host integration-cumin-01 is UP: PING OK - Packet loss = 0%, RTA = 1.70 ms [22:56:19] RECOVERY - SSH on integration-cumin-01 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [22:57:51] (03PS1) 10Krinkle: Add Jenkins jobs for labs/tools/nagf [integration/config] - 10https://gerrit.wikimedia.org/r/617276 [22:58:04] (03CR) 10Krinkle: [C: 03+2] Add Jenkins jobs for labs/tools/nagf [integration/config] - 10https://gerrit.wikimedia.org/r/617276 (owner: 10Krinkle) [22:59:09] (03Merged) 10jenkins-bot: Add Jenkins jobs for labs/tools/nagf [integration/config] - 10https://gerrit.wikimedia.org/r/617276 (owner: 10Krinkle) [22:59:48] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/617276 [22:59:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:05:50] PROBLEM - Host integration-cumin-01 is DOWN: CRITICAL - Host Unreachable (172.16.1.185)