[00:02:31] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [00:02:31] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [00:38:59] 10Continuous-Integration-Config, 10AbuseFilter, 10CX-deployments: mediawiki/vendor REL1_31 no more ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10Legoktm) a:03Legoktm [00:38:59] 10Continuous-Integration-Config, 10AbuseFilter, 10CX-deployments: mediawiki/vendor REL1_31 no more ship dependencies for wmf extensions that are not in the mediawiki tarball - https://phabricator.wikimedia.org/T189560 (10Legoktm) a:03Legoktm [00:41:50] (03PS1) 10Legoktm: Restore wmf-quibble-* jobs for REL1_XX branches [integration/config] - 10https://gerrit.wikimedia.org/r/473311 (https://phabricator.wikimedia.org/T189560) [00:41:50] (03PS1) 10Legoktm: Restore wmf-quibble-* jobs for REL1_XX branches [integration/config] - 10https://gerrit.wikimedia.org/r/473311 (https://phabricator.wikimedia.org/T189560) [00:44:19] (03CR) 10jerkins-bot: [V: 04-1] Restore wmf-quibble-* jobs for REL1_XX branches [integration/config] - 10https://gerrit.wikimedia.org/r/473311 (https://phabricator.wikimedia.org/T189560) (owner: 10Legoktm) [00:44:19] (03CR) 10jerkins-bot: [V: 04-1] Restore wmf-quibble-* jobs for REL1_XX branches [integration/config] - 10https://gerrit.wikimedia.org/r/473311 (https://phabricator.wikimedia.org/T189560) (owner: 10Legoktm) [00:52:36] (03PS1) 10Legoktm: Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) [00:52:36] (03PS1) 10Legoktm: Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) [00:52:44] legoktm: Gosh. [00:52:44] legoktm: Gosh. [00:53:07] it's happening \o/ [00:53:07] it's happening \o/ [00:53:30] Well, given we're running 7.2 on mwdebug in prod, yes, yes it is. :-) [00:53:30] Well, given we're running 7.2 on mwdebug in prod, yes, yes it is. :-) [00:53:49] heh [00:53:49] heh [00:53:58] Also I have 17% packet loss to Gerrit :< [00:53:58] Also I have 17% packet loss to Gerrit :< [00:54:03] Erk. [00:54:03] Erk. [00:54:09] 3G on CalTrain? [00:54:09] 3G on CalTrain? [00:54:20] literally sitting at home on my home wifi [00:54:20] literally sitting at home on my home wifi [00:54:40] Huh. [00:54:41] Huh. [00:55:47] (03CR) 10jerkins-bot: [V: 04-1] Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [00:55:47] (03CR) 10jerkins-bot: [V: 04-1] Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [00:57:05] (03PS2) 10Legoktm: Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) [00:57:05] (03PS2) 10Legoktm: Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) [00:57:53] (03CR) 10Legoktm: [C: 032] Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [00:57:53] (03CR) 10Legoktm: [C: 032] Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [00:59:36] (03Merged) 10jenkins-bot: Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [00:59:36] (03Merged) 10jenkins-bot: Make PHP 7.2 MediaWiki tests voting [integration/config] - 10https://gerrit.wikimedia.org/r/473312 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [00:59:39] 16:58:51 PHP Warning: Declaration of SpecialPageTranslationMovePage::showForm($par) should be compatible with MovePageForm::showForm($err, $isPermError = false) in /workspace/src/extensions/Translate/tag/SpecialPageTranslationMovePage.php on line 16 [00:59:39] 16:58:51 PHP Warning: Declaration of SpecialPageTranslationMovePage::showForm($par) should be compatible with MovePageForm::showForm($err, $isPermError = false) in /workspace/src/extensions/Translate/tag/SpecialPageTranslationMovePage.php on line 16 [00:59:39] 16:58:51 [00:59:39] 16:58:51 Warning: Declaration of SpecialPageTranslationMovePage::showForm($par) should be compatible with MovePageForm::showForm($err, $isPermError = false) in /workspace/src/extensions/Translate/tag/SpecialPageTranslationMovePage.php on line 16 [00:59:39] 16:58:51 [00:59:39] 16:58:51 Warning: Declaration of SpecialPageTranslationMovePage::showForm($par) should be compatible with MovePageForm::showForm($err, $isPermError = false) in /workspace/src/extensions/Translate/tag/SpecialPageTranslationMovePage.php on line 16 [01:39:58] 10MediaWiki-Codesniffer, 10Patch-For-Review: Allow deprecated @type in MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203922 (10Legoktm) Current usage of `@type` (with some false positives): https://codesearch.wmflabs.org/search/?q=%40type&i=nope&files=%5C.... [01:39:58] 10MediaWiki-Codesniffer, 10Patch-For-Review: Allow deprecated @type in MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203922 (10Legoktm) Current usage of `@type` (with some false positives): https://codesearch.wmflabs.org/search/?q=%40type&i=nope&files=%5C.... [01:41:28] (03CR) 10Legoktm: [C: 032] Remove [optional] from types in @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/472525 (https://phabricator.wikimedia.org/T196773) (owner: 10Umherirrender) [01:41:28] (03CR) 10Legoktm: [C: 032] Remove [optional] from types in @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/472525 (https://phabricator.wikimedia.org/T196773) (owner: 10Umherirrender) [01:42:45] (03Merged) 10jenkins-bot: Remove [optional] from types in @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/472525 (https://phabricator.wikimedia.org/T196773) (owner: 10Umherirrender) [01:42:45] (03Merged) 10jenkins-bot: Remove [optional] from types in @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/472525 (https://phabricator.wikimedia.org/T196773) (owner: 10Umherirrender) [01:43:16] (03CR) 10jenkins-bot: Remove [optional] from types in @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/472525 (https://phabricator.wikimedia.org/T196773) (owner: 10Umherirrender) [01:43:16] (03CR) 10jenkins-bot: Remove [optional] from types in @param [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/472525 (https://phabricator.wikimedia.org/T196773) (owner: 10Umherirrender) [01:45:29] (03PS1) 10Legoktm: Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 [01:45:29] (03PS1) 10Legoktm: Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 [01:45:45] 10Phabricator: Changing task title from empty to non-empty shows as "created this task" in history - https://phabricator.wikimedia.org/T209449 (10matmarex) [01:45:45] 10Phabricator: Changing task title from empty to non-empty shows as "created this task" in history - https://phabricator.wikimedia.org/T209449 (10matmarex) [01:51:28] (03CR) 10Legoktm: [C: 032] Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 (owner: 10Legoktm) [01:51:28] (03CR) 10Legoktm: [C: 032] Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 (owner: 10Legoktm) [01:52:23] (03Merged) 10jenkins-bot: Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 (owner: 10Legoktm) [01:52:23] (03Merged) 10jenkins-bot: Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 (owner: 10Legoktm) [01:52:55] (03CR) 10jenkins-bot: Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 (owner: 10Legoktm) [01:52:55] (03CR) 10jenkins-bot: Release 23.0.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/473316 (owner: 10Legoktm) [04:09:06] Project mediawiki-core-doxygen-docker build #2470: 04FAILURE in 5 min 3 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/2470/ [04:09:06] Project mediawiki-core-doxygen-docker build #2470: 04FAILURE in 5 min 3 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/2470/ [04:33:38] Project beta-code-update-eqiad build #224974: 15ABORTED in 30 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224974/ [04:33:38] Project beta-code-update-eqiad build #224974: 15ABORTED in 30 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224974/ [04:53:48] Project beta-code-update-eqiad build #224975: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224975/ [04:53:48] Project beta-code-update-eqiad build #224975: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224975/ [05:09:02] Project mediawiki-core-doxygen-docker build #2471: 04STILL FAILING in 5 min 0 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/2471/ [05:09:02] Project mediawiki-core-doxygen-docker build #2471: 04STILL FAILING in 5 min 0 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/2471/ [05:17:40] Project beta-code-update-eqiad build #224976: 04STILL FAILING in 23 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224976/ [05:17:40] Project beta-code-update-eqiad build #224976: 04STILL FAILING in 23 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224976/ [05:21:51] failure?? [05:21:51] failure?? [05:22:13] 04:04:02 > git fetch --no-tags --progress https://gerrit.wikimedia.org/r/p/mediawiki/core.git +refs/heads/*:refs/remotes/origin/* [05:22:13] 04:04:02 > git fetch --no-tags --progress https://gerrit.wikimedia.org/r/p/mediawiki/core.git +refs/heads/*:refs/remotes/origin/* [05:22:13] 04:09:05 ERROR: Error cloning remote repo 'origin' [05:22:13] 04:09:05 ERROR: Error cloning remote repo 'origin' [05:22:13] 04:09:05 hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https://gerrit.wikimedia.org/r/p/mediawiki/core.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: [05:22:13] 04:09:05 hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https://gerrit.wikimedia.org/r/p/mediawiki/core.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: [05:22:13] 04:09:05 stdout: [05:22:13] 04:09:05 stdout: [05:22:13] 04:09:05 stderr: fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/core.git/': Operation timed out after 0 milliseconds with 0 out of 0 bytes received [05:22:13] 04:09:05 stderr: fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/core.git/': Operation timed out after 0 milliseconds with 0 out of 0 bytes received [05:36:47] Project beta-code-update-eqiad build #224977: 04STILL FAILING in 19 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224977/ [05:36:47] Project beta-code-update-eqiad build #224977: 04STILL FAILING in 19 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224977/ [05:38:22] Project beta-code-update-eqiad build #224978: 04STILL FAILING in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224978/ [05:38:22] Project beta-code-update-eqiad build #224978: 04STILL FAILING in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224978/ [05:42:03] 10Gerrit: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Legoktm) p:05Triage>03Unbreak! [05:42:03] 10Gerrit: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Legoktm) p:05Triage>03Unbreak! [05:43:05] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [05:43:05] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [05:44:13] 10Gerrit: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) a:03Joe [05:44:13] 10Gerrit: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) a:03Joe [05:44:27] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) [05:44:27] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) [05:44:32] Project beta-code-update-eqiad build #224979: 04STILL FAILING in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224979/ [05:44:32] Project beta-code-update-eqiad build #224979: 04STILL FAILING in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224979/ [05:49:44] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) I was still quite asleep, but I saw a series of broken pipes from sockets and jetty refusing to manage any new connection in the logs, so I just restarted gerrit. It is now working, so we can lower the... [05:49:44] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) I was still quite asleep, but I saw a series of broken pipes from sockets and jetty refusing to manage any new connection in the logs, so I just restarted gerrit. It is now working, so we can lower the... [05:49:54] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) p:05Unbreak!>03High [05:49:54] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) p:05Unbreak!>03High [05:54:33] Yippee, build fixed! [05:54:34] Yippee, build fixed! [05:54:34] Project beta-code-update-eqiad build #224980: 09FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224980/ [05:54:34] Project beta-code-update-eqiad build #224980: 09FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/224980/ [06:02:51] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Legoktm) [21:45:43] I didn't realize it was upgraded today, I had been encountering weird behavior a few hours ago that I wasn't sure about ... [21:58:11] I couldn't get https://gerrit.wi... [06:02:52] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Legoktm) [21:45:43] I didn't realize it was upgraded today, I had been encountering weird behavior a few hours ago that I wasn't sure about ... [21:58:11] I couldn't get https://gerrit.wi... [06:03:31] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:03:31] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:11:27] Yippee, build fixed! [06:11:27] Yippee, build fixed! [06:11:27] Project mediawiki-core-doxygen-docker build #2472: 09FIXED in 7 min 25 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/2472/ [06:11:27] Project mediawiki-core-doxygen-docker build #2472: 09FIXED in 7 min 25 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/2472/ [08:23:17] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) a:05Joe>03None [08:23:17] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10Joe) a:05Joe>03None [08:25:38] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [08:25:38] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [09:09:40] 10MediaWiki-Codesniffer, 10Patch-For-Review: Allow deprecated @type in MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203922 (10thiemowmde) I honestly can't find a single valid usage of `@type` in this search result. https://gerrit.wikimedia.org/r/plugins/... [09:09:40] 10MediaWiki-Codesniffer, 10Patch-For-Review: Allow deprecated @type in MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203922 (10thiemowmde) I honestly can't find a single valid usage of `@type` in this search result. https://gerrit.wikimedia.org/r/plugins/... [09:53:11] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10WMDE-Fisch) a:05WMDE-Fisch>03None Wait, why was this assigned to me? o.O [09:53:11] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10WMDE-Fisch) a:05WMDE-Fisch>03None Wait, why was this assigned to me? o.O [10:00:16] ^ lol [10:00:17] ^ lol [10:00:53] (03CR) 10Hashar: [C: 04-1] "The wmf-quibble jobs are intended for Wikimedia deployment and thus solely for the master and wmf branches." [integration/config] - 10https://gerrit.wikimedia.org/r/473311 (https://phabricator.wikimedia.org/T189560) (owner: 10Legoktm) [10:00:54] (03CR) 10Hashar: [C: 04-1] "The wmf-quibble jobs are intended for Wikimedia deployment and thus solely for the master and wmf branches." [integration/config] - 10https://gerrit.wikimedia.org/r/473311 (https://phabricator.wikimedia.org/T189560) (owner: 10Legoktm) [10:08:01] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10hashar) I do not have any available bandwidth for the next 4 weeks at least, sorry. I am running the MediaWiki train from 11/12 to 11/16 and... [10:08:01] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Cloud-Services, 10Epic: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10hashar) I do not have any available bandwidth for the next 4 weeks at least, sorry. I am running the MediaWiki train from 11/12 to 11/16 and... [10:16:17] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10hashar) [10:16:17] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10hashar) [10:32:37] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (101Veertje) Adding the goat token was assigned to you, so I figured you knew how to add tokens https://phabricator.wikimedia.org/T173130 [10:32:37] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (101Veertje) Adding the goat token was assigned to you, so I figured you knew how to add tokens https://phabricator.wikimedia.org/T173130 [10:33:04] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10hashar) That might be related to Gerrit 2.15.6 upgrade T205784 . I am not familiar with Jetty though but we can at least dig in the logs on the cobalt server. [10:33:04] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10hashar) That might be related to Gerrit 2.15.6 upgrade T205784 . I am not familiar with Jetty though but we can at least dig in the logs on the cobalt server. [10:39:47] !log added gtirloni as projectadmin in deployment-prep project [10:39:47] !log added gtirloni as projectadmin in deployment-prep project [10:39:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:39:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:48:20] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10hashar) We had a lot of stack overflow errors on random pages such as gittiles, that is due to some pages being very long to prettify. Then at 5:36:43 UTC we had a lot of: ` [2018-11-14 05:36:43,643] [HTTP-8... [10:48:20] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10hashar) We had a lot of stack overflow errors on random pages such as gittiles, that is due to some pages being very long to prettify. Then at 5:36:43 UTC we had a lot of: ` [2018-11-14 05:36:43,643] [HTTP-8... [10:52:49] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10Aklapper) I don't think I understand the use case? If people write `bug:T12345` then the patch in Gerrit will be linked from `T12345` anyway. Where to add what exactly? [10:52:49] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10Aklapper) I don't think I understand the use case? If people write `bug:T12345` then the patch in Gerrit will be linked from `T12345` anyway. Where to add what exactly? [10:54:33] 10Phabricator (Upstream), 10Upstream: Changing task title from empty to non-empty shows as "created this task" in history - https://phabricator.wikimedia.org/T209449 (10Aklapper) [10:54:33] 10Phabricator (Upstream), 10Upstream: Changing task title from empty to non-empty shows as "created this task" in history - https://phabricator.wikimedia.org/T209449 (10Aklapper) [10:54:53] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: deployment-prep: Hosts haven't applied new firewall rules - https://phabricator.wikimedia.org/T209466 (10GTirloni) [10:54:53] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: deployment-prep: Hosts haven't applied new firewall rules - https://phabricator.wikimedia.org/T209466 (10GTirloni) [11:01:49] 10Phabricator (2017-09-06), 10Goatification: Add Phabricator token "goat" / "meh" - https://phabricator.wikimedia.org/T173130 (10WMDE-Fisch) a:05WMDE-Fisch>03None [11:01:49] 10Phabricator (2017-09-06), 10Goatification: Add Phabricator token "goat" / "meh" - https://phabricator.wikimedia.org/T173130 (10WMDE-Fisch) a:05WMDE-Fisch>03None [11:03:19] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10WMDE-Fisch) >>! In T208760#4745672, @1Veertje wrote: > Adding the goat token was assigned to you, so I figured you knew how to add tokens > > https://phabricator.wikimedia.org/T173130 I see :-) - no I have no sor... [11:03:19] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10WMDE-Fisch) >>! In T208760#4745672, @1Veertje wrote: > Adding the goat token was assigned to you, so I figured you knew how to add tokens > > https://phabricator.wikimedia.org/T173130 I see :-) - no I have no sor... [11:18:04] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10TheDJ) This seems like something for #phabricator admins. I'm not aware of an acl groups for token management and it doesn't seem part of project-admins, so i assume @mmodell or @demon as actually administrators mig... [11:18:04] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10TheDJ) This seems like something for #phabricator admins. I'm not aware of an acl groups for token management and it doesn't seem part of project-admins, so i assume @mmodell or @demon as actually administrators mig... [11:25:38] 10Release-Engineering-Team, 10Operations, 10Release Pipeline: Design pipeline image versioning scheme - https://phabricator.wikimedia.org/T209088 (10akosiaris) I think we should support multiple tags per image (docker anyway does support that and they cost next to nothing on the registry level AFAIK) * Keep... [11:25:38] 10Release-Engineering-Team, 10Operations, 10Release Pipeline: Design pipeline image versioning scheme - https://phabricator.wikimedia.org/T209088 (10akosiaris) I think we should support multiple tags per image (docker anyway does support that and they cost next to nothing on the registry level AFAIK) * Keep... [11:36:11] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Aklapper) Heh, that's the Wikimedia Commons discussion whether to create lots of very specific categories such as `Female Australian writers living in Estonia and speaking Chinese`, or to expect folks to se... [11:36:11] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Aklapper) Heh, that's the Wikimedia Commons discussion whether to create lots of very specific categories such as `Female Australian writers living in Estonia and speaking Chinese`, or to expect folks to se... [11:38:00] Oh wow gerrit managed to be exhausted? [11:38:00] Oh wow gerrit managed to be exhausted? [12:11:53] 10Release-Engineering-Team, 10Operations, 10Release Pipeline: Design pipeline image versioning scheme - https://phabricator.wikimedia.org/T209088 (10fselles) +1 latest should be avoided for production, in my experience is also problematic for development (since you don't know which version are you running c... [12:11:53] 10Release-Engineering-Team, 10Operations, 10Release Pipeline: Design pipeline image versioning scheme - https://phabricator.wikimedia.org/T209088 (10fselles) +1 latest should be avoided for production, in my experience is also problematic for development (since you don't know which version are you running c... [13:28:10] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10D3r1ck01) [13:28:11] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10D3r1ck01) [13:28:28] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10D3r1ck01) [13:28:28] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10D3r1ck01) [13:32:26] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10D3r1ck01) [13:32:26] 10MediaWiki-Codesniffer: PHP Codesniffer errors and warnings misleading(?) - https://phabricator.wikimedia.org/T209486 (10D3r1ck01) [13:59:43] PROBLEM - Puppet staleness on deployment-rd3-cptest-master01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [13:59:43] PROBLEM - Puppet staleness on deployment-rd3-cptest-master01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [14:11:23] (03PS1) 10Ema: Test fifo-log-demux with debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/473503 (https://phabricator.wikimedia.org/T204225) [14:11:23] (03PS1) 10Ema: Test fifo-log-demux with debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/473503 (https://phabricator.wikimedia.org/T204225) [14:30:04] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T206658 (10hashar) Train on hold based on T209429 . I have posted to wikitech-l about it [14:30:04] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T206658 (10hashar) Train on hold based on T209429 . I have posted to wikitech-l about it [15:23:24] PROBLEM - Host integration-slave-docker-1021 is DOWN: CRITICAL - Host Unreachable (10.68.22.8) [15:23:24] PROBLEM - Host integration-slave-docker-1021 is DOWN: CRITICAL - Host Unreachable (10.68.22.8) [15:54:22] (03CR) 10Zfilipin: [C: 031] "This just adds description to the job?" [integration/config] - 10https://gerrit.wikimedia.org/r/469906 (owner: 10Hashar) [15:54:22] (03CR) 10Zfilipin: [C: 031] "This just adds description to the job?" [integration/config] - 10https://gerrit.wikimedia.org/r/469906 (owner: 10Hashar) [16:01:38] PROBLEM - SSH on integration-slave-jessie-1001 is CRITICAL: connect to address 10.68.16.72 and port 22: No route to host [16:01:38] PROBLEM - SSH on integration-slave-jessie-1001 is CRITICAL: connect to address 10.68.16.72 and port 22: No route to host [16:03:06] PROBLEM - Host integration-slave-jessie-1001 is DOWN: CRITICAL - Host Unreachable (10.68.16.72) [16:03:06] PROBLEM - Host integration-slave-jessie-1001 is DOWN: CRITICAL - Host Unreachable (10.68.16.72) [16:06:38] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10thcipriani) >>! In T209456#4745707, @hashar wrote: > We had a lot of stack overflow errors on random pages such as gittiles, that is due to some pages being very long to prettify. Those happened with some f... [16:06:38] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10thcipriani) >>! In T209456#4745707, @hashar wrote: > We had a lot of stack overflow errors on random pages such as gittiles, that is due to some pages being very long to prettify. Those happened with some f... [16:13:26] (03CR) 10Zfilipin: [C: 04-1] "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:13:26] (03CR) 10Zfilipin: [C: 04-1] "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:14:23] (03CR) 10Zfilipin: [C: 04-1] "mediawiki-core-qunit-selenium-jessie is running in experimental pipeline, I've just started it for https://gerrit.wikimedia.org/r/c/mediaw" [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:14:23] (03CR) 10Zfilipin: [C: 04-1] "mediawiki-core-qunit-selenium-jessie is running in experimental pipeline, I've just started it for https://gerrit.wikimedia.org/r/c/mediaw" [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:15:54] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), 10Patch-For-Review: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 (10kostajh) a:05kostajh>03None Unassigning myself for now. When coming back to this I'd like to check... [16:15:54] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), 10Patch-For-Review: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 (10kostajh) a:05kostajh>03None Unassigning myself for now. When coming back to this I'd like to check... [16:31:18] (03CR) 10Zfilipin: [C: 04-1] "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:31:18] (03CR) 10Zfilipin: [C: 04-1] "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:31:53] (03CR) 10Zfilipin: "This should probably just be abandoned." [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:31:53] (03CR) 10Zfilipin: "This should probably just be abandoned." [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [16:57:58] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) How about this one? {F27221251} Resized and cropped from https://commons.wikimedia.org/wiki/File:Eenstroopwafel.jpg [16:57:59] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) How about this one? {F27221251} Resized and cropped from https://commons.wikimedia.org/wiki/File:Eenstroopwafel.jpg [16:59:37] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) At 18px I'm not sure it's as recognizable: {F27221276} [16:59:37] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) At 18px I'm not sure it's as recognizable: {F27221276} [17:00:53] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) As for adding them, it's done by adding to the sprite sheet in the phabricator source. [17:00:53] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) As for adding them, it's done by adding to the sprite sheet in the phabricator source. [17:05:01] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:05:01] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:08:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10greg) [17:08:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10greg) [17:11:48] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10Wikibase-Quality, and 2 others: WBQC gate-and-submit fails due to deprecated function in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) Adding #continuous-integration-config because quibbl... [17:11:48] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10Wikibase-Quality, and 2 others: WBQC gate-and-submit fails due to deprecated function in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) Adding #continuous-integration-config because quibbl... [17:12:24] fyi upstream are removing GWTUI today. [17:12:24] fyi upstream are removing GWTUI today. [17:12:27] (gerrit) [17:12:27] (gerrit) [17:18:52] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10Wikibase-Quality, and 2 others: WBQC gate-and-submit fails due to deprecated function in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) Apparently, in mediawiki-vendor a patch was backport... [17:18:52] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10Wikibase-Quality, and 2 others: WBQC gate-and-submit fails due to deprecated function in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) Apparently, in mediawiki-vendor a patch was backport... [17:20:07] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10Wikibase-Quality, and 2 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) [17:20:07] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10Wikibase-Quality, and 2 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) [17:26:13] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) [17:26:13] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Lucas_Werkmeister_WMDE) [17:49:43] (03PS1) 10Hashar: Skip php 7.1 and 7.2 on wmf/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/473571 [17:49:43] (03PS1) 10Hashar: Skip php 7.1 and 7.2 on wmf/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/473571 [17:49:45] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Ladsgroup) Property Sugg... [17:49:45] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Ladsgroup) Property Sugg... [17:50:41] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10Phil_Boswell) Shouldn't it really be SVG to allow for scaling? [17:50:41] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10Phil_Boswell) Shouldn't it really be SVG to allow for scaling? [17:52:12] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) phabricator sprite sheets don't support svg. It has to be png in exactly two sizes: `18x18 pixels` and `36x36 pixels` [17:52:12] 10Phabricator: Add phabricator token "stroopwafel" - https://phabricator.wikimedia.org/T208760 (10mmodell) phabricator sprite sheets don't support svg. It has to be png in exactly two sizes: `18x18 pixels` and `36x36 pixels` [17:53:04] and GWTUI has now been removed :) [17:53:04] and GWTUI has now been removed :) [17:53:09] https://gerrit-review.googlesource.com/c/gerrit/+/116790 [17:53:09] https://gerrit-review.googlesource.com/c/gerrit/+/116790 [17:54:11] (03CR) 10Hashar: "At least tests are passing. Gotta carefully review the diff, and maybe I will add a test for the wmf branches, we already have to exhausti" [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [17:54:11] (03CR) 10Hashar: "At least tests are passing. Gotta carefully review the diff, and maybe I will add a test for the wmf branches, we already have to exhausti" [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [17:58:19] 10Gerrit: Gerrit word diff is really bad at matching - https://phabricator.wikimedia.org/T181961 (10Paladox) Upstream removed GWTUI (the old ui). I propose to decline this (or does it do the same in the new ui too?) [17:58:19] 10Gerrit: Gerrit word diff is really bad at matching - https://phabricator.wikimedia.org/T181961 (10Paladox) Upstream removed GWTUI (the old ui). I propose to decline this (or does it do the same in the new ui too?) [18:00:18] 10Gerrit: Gerrit CoC and Privacy Policy links showing up in an awkward place when editing a file - https://phabricator.wikimedia.org/T206530 (10Paladox) Upstream removed GWTUI today (the old ui). I propose to decline this (or does it do the same in the new ui too?) [18:00:18] 10Gerrit: Gerrit CoC and Privacy Policy links showing up in an awkward place when editing a file - https://phabricator.wikimedia.org/T206530 (10Paladox) Upstream removed GWTUI today (the old ui). I propose to decline this (or does it do the same in the new ui too?) [18:03:01] Krenair et al, today's list of servers-andrew-wants-to-move-out-of-the-way is: deployment-chromium01, deployment-mediawiki-07, deployment-mx02, deployment-snapshot01 [18:03:01] Krenair et al, today's list of servers-andrew-wants-to-move-out-of-the-way is: deployment-chromium01, deployment-mediawiki-07, deployment-mx02, deployment-snapshot01 [18:03:06] Anything time-critical in there? [18:03:06] Anything time-critical in there? [18:03:35] I'd check with Ariel about snapshot01, I don't really know how that works [18:03:36] I'd check with Ariel about snapshot01, I don't really know how that works [18:03:51] mx being out may loose some emails but probably not a huge deal [18:03:52] mx being out may loose some emails but probably not a huge deal [18:04:13] apergos: ok if deployment-snapshot01 has some downtime? [18:04:13] apergos: ok if deployment-snapshot01 has some downtime? [18:04:22] feel free [18:04:22] feel free [18:04:24] (maybe a lot of downtime if it actually contains snapshots) [18:04:24] (maybe a lot of downtime if it actually contains snapshots) [18:04:27] great, thanks [18:04:27] great, thanks [18:04:40] it will have only teeny tiny dump fles on it [18:04:40] it will have only teeny tiny dump fles on it [18:04:42] mediawiki-07 is used by services (I guess RB, parsoid and co.?) to hit the MW API so may be important for serving requests [18:04:42] nothing huge [18:04:42] nothing huge [18:04:42] mediawiki-07 is used by services (I guess RB, parsoid and co.?) to hit the MW API so may be important for serving requests [18:04:58] could probably shift that to using a different MW server in the mean time though [18:04:59] could probably shift that to using a different MW server in the mean time though [18:05:16] don't remember what chromium does [18:05:16] don't remember what chromium does [18:05:26] oh it's proton [18:05:26] oh it's proton [18:06:30] maybe just check with someone like mobrovac about that one? [18:06:30] maybe just check with someone like mobrovac about that one? [18:08:21] andrewbogott, want me to try to redirect services at a different MW host? [18:08:21] andrewbogott, want me to try to redirect services at a different MW host? [18:08:43] for mediawiki-07? Sure. [18:08:43] for mediawiki-07? Sure. [18:08:55] yeah [18:08:55] yeah [18:09:45] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10mmodell) So the problem seems to have started between 02:00 and 02:12 UTC. There was a fairly large spike in outgoing traffic on eth0 between 02:10 and 02:12 at which point cpu load gradually falls off as a... [18:09:45] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10mmodell) So the problem seems to have started between 02:00 and 02:12 UTC. There was a fairly large spike in outgoing traffic on eth0 between 02:10 and 02:12 at which point cpu load gradually falls off as a... [18:10:46] Project beta-scap-eqiad build #227740: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227740/ [18:10:46] Project beta-scap-eqiad build #227740: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227740/ [18:16:22] (03CR) 10Legoktm: [C: 04-1] "7.1 should be skipped (I thought we already had a branch filter for it?), but 7.2 needs to be run for wmf/ since that's what we will be an" [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [18:16:22] (03CR) 10Legoktm: [C: 04-1] "7.1 should be skipped (I thought we already had a branch filter for it?), but 7.2 needs to be run for wmf/ since that's what we will be an" [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [18:16:34] huh some stuff is still going to mediawiki 07 [18:16:34] huh some stuff is still going to mediawiki 07 [18:17:43] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10mmodell) @thcipriani is going to try installing https://gerrit-review.googlesource.com/admin/repos/plugins/javamelody to hopefully collect some more useful data about the state of the JVM. At this point we... [18:17:43] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10mmodell) @thcipriani is going to try installing https://gerrit-review.googlesource.com/admin/repos/plugins/javamelody to hopefully collect some more useful data about the state of the JVM. At this point we... [18:20:35] RECOVERY - Host deployment-snapshot01 is UP: PING OK - Packet loss = 0%, RTA = 2.44 ms [18:20:35] RECOVERY - Host deployment-snapshot01 is UP: PING OK - Packet loss = 0%, RTA = 2.44 ms [18:21:20] this doesn't make sense [18:21:20] this doesn't make sense [18:22:53] my own requests seem to be handled by mediawiki-07 but varnish doesn't have a backend for it [18:22:53] my own requests seem to be handled by mediawiki-07 but varnish doesn't have a backend for it [18:23:34] that's surprising [18:23:34] that's surprising [18:25:05] what do you think about chromium01? It's hard for me to imagine that pdf rendering is happening 24/7... [18:25:05] what do you think about chromium01? It's hard for me to imagine that pdf rendering is happening 24/7... [18:26:00] Yippee, build fixed! [18:26:00] Yippee, build fixed! [18:26:01] Project beta-scap-eqiad build #227741: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227741/ [18:26:01] Project beta-scap-eqiad build #227741: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227741/ [18:26:14] Waiting for marco is fine, I'm just on a roll :) [18:26:14] Waiting for marco is fine, I'm just on a roll :) [18:26:58] andrewbogott, it's probably fine [18:26:59] andrewbogott, it's probably fine [18:32:11] 10Project-Admins: Service Operations kanban workboard request - https://phabricator.wikimedia.org/T209522 (10jijiki) p:05Triage>03Normal [18:32:11] 10Project-Admins: Service Operations kanban workboard request - https://phabricator.wikimedia.org/T209522 (10jijiki) p:05Triage>03Normal [18:34:54] PROBLEM - Host deployment-chromium01 is DOWN: CRITICAL - Host Unreachable (10.68.19.44) [18:34:54] PROBLEM - Host deployment-chromium01 is DOWN: CRITICAL - Host Unreachable (10.68.19.44) [18:36:55] oh there it is [18:36:55] oh there it is [18:41:57] yeah so it was in the hiera data in horizon for the deployment-cache-text* prefix [18:41:57] yeah so it was in the hiera data in horizon for the deployment-cache-text* prefix [18:42:24] appservers and appservers_debug [18:42:24] appservers and appservers_debug [18:45:53] andrewbogott, should be fine to move mediawiki-07 now [18:45:53] andrewbogott, should be fine to move mediawiki-07 now [18:46:04] /var/log/apache2/other_vhosts_access.log isn't getting anything new [18:46:04] /var/log/apache2/other_vhosts_access.log isn't getting anything new [18:47:36] RECOVERY - Host deployment-chromium01 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms [18:47:36] RECOVERY - Host deployment-chromium01 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms [18:48:12] Krenair: thanks! [18:48:12] Krenair: thanks! [18:48:41] turns out I was looking for the wrong stuff in the wrong places before [18:48:41] turns out I was looking for the wrong stuff in the wrong places before [18:53:13] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) >>! In... [18:53:14] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) >>! In... [18:53:42] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) p:05Tr... [18:53:42] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) p:05Tr... [18:53:46] PROBLEM - Host deployment-mediawiki-07 is DOWN: CRITICAL - Host Unreachable (10.68.18.62) [18:53:46] PROBLEM - Host deployment-mediawiki-07 is DOWN: CRITICAL - Host Unreachable (10.68.18.62) [18:53:54] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) CC @Leg... [18:53:54] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) CC @Leg... [18:55:13] Project beta-scap-eqiad build #227743: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227743/ [18:55:13] Project beta-scap-eqiad build #227743: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227743/ [18:56:35] oh yeah scap isn't 100% happy with that [18:56:35] oh yeah scap isn't 100% happy with that [18:56:38] but it still runs so okay [18:56:38] but it still runs so okay [18:57:18] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10hashar) For monitoring it seems we monitor the JVM with JMX. There is a task for Gerrit at: T184086 The CI Jenkins used to have Java melody but that is apparently no more enabled. [18:57:18] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10hashar) For monitoring it seems we monitor the JVM with JMX. There is a task for Gerrit at: T184086 The CI Jenkins used to have Java melody but that is apparently no more enabled. [19:06:16] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10hashar) I thought about a link in the left side bar such as `Search Gerrit`. Some tasks have lot of patchsets and it is a bit hard to keep track of which are still pending. I then end... [19:06:16] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10hashar) I thought about a link in the left side bar such as `Search Gerrit`. Some tasks have lot of patchsets and it is a bit hard to keep track of which are still pending. I then end... [19:09:12] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Legoktm) * Either we swi... [19:09:12] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Legoktm) * Either we swi... [19:09:45] Project beta-scap-eqiad build #227744: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227744/ [19:09:45] Project beta-scap-eqiad build #227744: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227744/ [19:09:52] legoktm: Oh, right, these extensions do their own thing for no good reason. :-( [19:09:53] legoktm: Oh, right, these extensions do their own thing for no good reason. :-( [19:13:39] YEAH :< [19:13:39] YEAH :< [19:13:46] sorry caps lock [19:13:46] sorry caps lock [19:14:03] also, now that we're running PHP 7.2 in production, disabling CI isn't an option [19:14:03] also, now that we're running PHP 7.2 in production, disabling CI isn't an option [19:21:47] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T206658 (10hashar) [19:21:47] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T206658 (10hashar) [19:23:27] legoktm: if you merge this i imagine the rest will fix itself: https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/472160/ [19:23:27] legoktm: if you merge this i imagine the rest will fix itself: https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/472160/ [19:24:08] Project beta-scap-eqiad build #227745: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227745/ [19:24:08] Project beta-scap-eqiad build #227745: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227745/ [19:24:28] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Operations: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10thcipriani) Seems like the metrics reporter plugin hasn't received any updates for 8 months now: https://gerrit.googlesource.com/plugins/metrics-reporter-... [19:24:28] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Operations: Add prometheus exporter to Gerrit - https://phabricator.wikimedia.org/T184086 (10thcipriani) Seems like the metrics reporter plugin hasn't received any updates for 8 months now: https://gerrit.googlesource.com/plugins/metrics-reporter-... [19:25:19] ebernhardson: how will that fix it? the problem is that those jobs aren't using mw/vendor, and are using composer instead. If it was using mw/vendor, then I already backported the don't-use-each() patch [19:25:19] ebernhardson: how will that fix it? the problem is that those jobs aren't using mw/vendor, and are using composer instead. If it was using mw/vendor, then I already backported the don't-use-each() patch [19:25:36] legoktm: once that is merged i can also merge the Elastica patch which updates composer.json [19:25:36] legoktm: once that is merged i can also merge the Elastica patch which updates composer.json [19:25:39] ahh ok [19:25:39] ahh ok [19:25:46] (reedy linked it in the comments section) [19:25:46] (reedy linked it in the comments section) [19:25:53] legoktm: what is the state of PHP on prod? Do we skip php7.0 entirely and just focus on hhvm/php7.2? [19:25:53] legoktm: what is the state of PHP on prod? Do we skip php7.0 entirely and just focus on hhvm/php7.2? [19:26:26] hashar: essentially, yes [19:26:26] hashar: essentially, yes [19:26:36] oh hm [19:26:37] oh hm [19:26:49] hashar: well, we're still running php7.0 in a few places like scap, but I think we'll get rid of that in a few weeks [19:26:49] hashar: well, we're still running php7.0 in a few places like scap, but I think we'll get rid of that in a few weeks [19:26:57] AH great [19:26:57] AH great [19:26:58] so [19:26:58] so [19:27:03] [HHVM, php7.2] [19:27:03] [HHVM, php7.2] [19:27:33] so I guess my Zuul config patch to drop out 7.1 and 7.2 for wm fbranches just have to be adjusted [19:27:33] so I guess my Zuul config patch to drop out 7.1 and 7.2 for wm fbranches just have to be adjusted [19:27:50] but I gotta review the diff, maybe add a test to ensure what is triggered for wmf branches [19:27:50] but I gotta review the diff, maybe add a test to ensure what is triggered for wmf branches [19:28:17] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10reosarevok) Our use case is tasks that are not time critical, but should be available there as something for volunteers to choose from if and when they have time (that is, that we do not prioritize for our... [19:28:17] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10reosarevok) Our use case is tasks that are not time critical, but should be available there as something for volunteers to choose from if and when they have time (that is, that we do not prioritize for our... [19:32:47] ebernhardson: +2'd [19:32:47] ebernhardson: +2'd [19:33:11] hashar: yeah, we can definitely ignore 7.1 [19:33:11] hashar: yeah, we can definitely ignore 7.1 [19:33:55] hashar: I was also thinking of having maybe the sqlite job run on 7.1, instead of having a mysql-7.1 and a sqlite-7.9 [19:33:55] hashar: I was also thinking of having maybe the sqlite job run on 7.1, instead of having a mysql-7.1 and a sqlite-7.9 [19:34:06] 7.0* [19:34:07] 7.0* [19:35:00] 10Gerrit, 10Operations: Add javamelody to gerrit - https://phabricator.wikimedia.org/T209526 (10thcipriani) p:05Triage>03High [19:35:00] 10Gerrit, 10Operations: Add javamelody to gerrit - https://phabricator.wikimedia.org/T209526 (10thcipriani) p:05Triage>03High [19:35:32] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: Add javamelody to gerrit - https://phabricator.wikimedia.org/T209526 (10thcipriani) a:03thcipriani [19:35:32] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: Add javamelody to gerrit - https://phabricator.wikimedia.org/T209526 (10thcipriani) a:03thcipriani [19:35:48] (03PS2) 10Hashar: Skip php 7.1 on wmf/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/473571 [19:35:48] (03PS2) 10Hashar: Skip php 7.1 on wmf/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/473571 [19:35:59] legoktm: thanks. I will review the zuul diff for that change later tonight [19:35:59] legoktm: thanks. I will review the zuul diff for that change later tonight [19:36:03] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Legoktm) a:03EBernhards... [19:36:03] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 3 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Legoktm) a:03EBernhards... [19:36:54] legoktm: for sqlite beware. It does not work at all on some REL branches [19:36:55] legoktm: for sqlite beware. It does not work at all on some REL branches [19:37:11] right, there's a branch filter so it only runs on 1.32+ IIRC [19:37:11] right, there's a branch filter so it only runs on 1.32+ IIRC [19:38:54] Yippee, build fixed! [19:38:54] Yippee, build fixed! [19:38:55] Project beta-scap-eqiad build #227746: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227746/ [19:38:55] Project beta-scap-eqiad build #227746: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/227746/ [19:48:46] legoktm: dpeloying the php7.1 exclusion for wmf/ branches. The diff looks good [19:48:46] legoktm: dpeloying the php7.1 exclusion for wmf/ branches. The diff looks good [19:49:04] seems multiple jobs: skip-if eventually get merged propery \o/ [19:49:04] seems multiple jobs: skip-if eventually get merged propery \o/ [19:49:36] woot :D [19:49:36] woot :D [19:49:44] then I guess I will tackle https://phabricator.wikimedia.org/T192702 : Skip Quibble "composer" and "sqlite" jobs on wmf branches [19:49:44] then I guess I will tackle https://phabricator.wikimedia.org/T192702 : Skip Quibble "composer" and "sqlite" jobs on wmf branches [19:50:13] (03CR) 10Hashar: [C: 032] "Diff seems ok. It seems skip-if from different jobs: entries get merged properly." [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [19:50:13] (03CR) 10Hashar: [C: 032] "Diff seems ok. It seems skip-if from different jobs: entries get merged properly." [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [19:52:39] (03Merged) 10jenkins-bot: Skip php 7.1 on wmf/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [19:52:39] (03Merged) 10jenkins-bot: Skip php 7.1 on wmf/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/473571 (owner: 10Hashar) [19:56:10] hashar: left a minor note on that task [19:56:10] hashar: left a minor note on that task [19:58:39] legoktm: noted :) [19:58:39] legoktm: noted :) [19:58:45] and seems the wmf patches also run release-quibble-vendor-mysql-php72-docker bah [19:58:45] and seems the wmf patches also run release-quibble-vendor-mysql-php72-docker bah [19:59:16] whaaat [19:59:16] whaaat [19:59:21] I swear we have branch filters for this [19:59:21] I swear we have branch filters for this [19:59:52] hashar: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/zuul/layout.yaml#988 [19:59:52] hashar: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/zuul/layout.yaml#988 [20:00:04] (03PS1) 10Hashar: wmf branches do not use composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/473593 [20:00:04] (03PS1) 10Hashar: wmf branches do not use composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/473593 [20:01:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10Quibble: Skip Quibble "composer" and "sqlite" jobs on wmf branches - https://phabricator.wikimedia.org/T192702 (10hashar) I went with a skip-if. If a job is matched by multiple jobs: filters each having a branch: , I think only the last... [20:01:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10Quibble: Skip Quibble "composer" and "sqlite" jobs on wmf branches - https://phabricator.wikimedia.org/T192702 (10hashar) I went with a skip-if. If a job is matched by multiple jobs: filters each having a branch: , I think only the last... [20:01:33] (03PS2) 10Hashar: wmf branches do not use composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/473593 (https://phabricator.wikimedia.org/T192702) [20:01:33] (03PS2) 10Hashar: wmf branches do not use composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/473593 (https://phabricator.wikimedia.org/T192702) [20:01:44] legoktm: yeah so [20:01:44] legoktm: yeah so [20:01:54] I think we have mutiple jobs filter matching that job [20:01:54] I think we have mutiple jobs filter matching that job [20:01:59] ughhh [20:01:59] ughhh [20:02:03] each having a different branch: stanza [20:02:03] each having a different branch: stanza [20:02:09] and probably only the last one get applied [20:02:09] and probably only the last one get applied [20:02:22] whereas mutliple skip-if get merged/aggregated properly [20:02:22] whereas mutliple skip-if get merged/aggregated properly [20:02:27] (that is my theory) [20:02:27] (that is my theory) [20:07:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10Quibble, 10Patch-For-Review: Skip Quibble "composer" and "sqlite" jobs on wmf branches - https://phabricator.wikimedia.org/T192702 (10hashar) For `release-quibble-vendor-mysql-php72-docker`, @legoktm mentioned it has: ` - name: ^relea... [20:07:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10Quibble, 10Patch-For-Review: Skip Quibble "composer" and "sqlite" jobs on wmf branches - https://phabricator.wikimedia.org/T192702 (10hashar) For `release-quibble-vendor-mysql-php72-docker`, @legoktm mentioned it has: ` - name: ^relea... [20:08:01] my assumption is on ^ ^ [20:08:01] my assumption is on ^ ^ [20:08:08] gotta run the train for group1 [20:08:08] gotta run the train for group1 [20:20:30] andrewbogott, so did all go well? [20:20:30] andrewbogott, so did all go well? [20:20:47] seems like — all done for now [20:20:47] seems like — all done for now [20:20:57] ok [20:20:57] ok [20:21:02] you got all the instances moved you needed to? [20:21:02] you got all the instances moved you needed to? [20:24:20] anyway moving traffic back to -mediawiki-07 [20:24:20] anyway moving traffic back to -mediawiki-07 [20:26:07] yeah, all moved, cleared out labvirt1015. [20:26:07] yeah, all moved, cleared out labvirt1015. [20:26:13] lol why is stuff still hitting bits.beta.wmflabs.org in 2018 [20:26:13] lol why is stuff still hitting bits.beta.wmflabs.org in 2018 [20:28:06] some UA involving IE8, AOL and Media Center PC, let's just ignore that one.. [20:28:06] some UA involving IE8, AOL and Media Center PC, let's just ignore that one.. [20:29:32] Krenair: Lol @ media center pc [20:29:33] Krenair: Lol @ media center pc [20:29:37] anyway mediawiki-07 is back [20:29:37] anyway mediawiki-07 is back [20:29:45] Is that the old media application in windows? [20:29:45] Is that the old media application in windows? [20:30:30] something like that [20:30:30] something like that [20:30:51] Ah ok [20:30:51] Ah ok [20:30:59] And also who uses IE8 these days??? [20:31:00] And also who uses IE8 these days??? [20:31:41] well quite [20:31:41] well quite [20:32:31] technically MW still has basic support for IE6+ [20:32:31] technically MW still has basic support for IE6+ [20:33:28] no JS there since 3 years ago though [20:33:28] no JS there since 3 years ago though [20:33:30] Yup [20:33:30] Yup [20:33:42] But why would anyone want to use ie8? [20:33:42] But why would anyone want to use ie8? [20:33:55] I haven’t used it in 8 or 10 years [20:33:55] I haven’t used it in 8 or 10 years [20:34:59] I'm guessing the people who run it still either don't have a choice in the matter or don't care [20:34:59] I'm guessing the people who run it still either don't have a choice in the matter or don't care [20:35:07] or don't know any better [20:35:07] or don't know any better [20:36:15] Ah ok [20:36:15] Ah ok [20:46:40] (03CR) 10Hashar: [C: 04-1] wmf branches do not use composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/473593 (https://phabricator.wikimedia.org/T192702) (owner: 10Hashar) [20:46:41] (03CR) 10Hashar: [C: 04-1] wmf branches do not use composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/473593 (https://phabricator.wikimedia.org/T192702) (owner: 10Hashar) [20:46:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10Quibble, 10Patch-For-Review: Skip Quibble "composer" and "sqlite" jobs on wmf branches - https://phabricator.wikimedia.org/T192702 (10hashar) The `skip-if` are not merged (just like `branch`), the last one in the config wins: ` lang=pyt... [20:46:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10Quibble, 10Patch-For-Review: Skip Quibble "composer" and "sqlite" jobs on wmf branches - https://phabricator.wikimedia.org/T192702 (10hashar) The `skip-if` are not merged (just like `branch`), the last one in the config wins: ` lang=pyt... [21:29:36] thcipriani hi, im wondering could we create a stable-2.16 branch at some point (so i can start testing with it) please? :) [21:29:36] thcipriani hi, im wondering could we create a stable-2.16 branch at some point (so i can start testing with it) please? :) [21:32:05] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 4 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Legoktm) 05Open>03Res... [21:32:05] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 4 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Legoktm) 05Open>03Res... [21:50:03] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10thcipriani) [21:50:03] 10Gerrit, 10Operations: Gerrit is down "502 Proxy Error" - https://phabricator.wikimedia.org/T209456 (10thcipriani) [21:50:09] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: Add javamelody to gerrit - https://phabricator.wikimedia.org/T209526 (10thcipriani) 05Open>03Resolved Now available for gerrit admins at: https://gerrit.wikimedia.org/r/monitoring [21:50:09] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: Add javamelody to gerrit - https://phabricator.wikimedia.org/T209526 (10thcipriani) 05Open>03Resolved Now available for gerrit admins at: https://gerrit.wikimedia.org/r/monitoring [21:58:12] https://twitter.com/GerritReview/status/1062824128271409152 [21:58:12] https://twitter.com/GerritReview/status/1062824128271409152 [21:58:37] thcipriani i've asked luca thorugh the hackathon channel if he knew how to debug (what you said in that thread) :) [21:58:37] thcipriani i've asked luca thorugh the hackathon channel if he knew how to debug (what you said in that thread) :) [21:59:05] paladox: cool, thank you! [21:59:05] paladox: cool, thank you! [21:59:16] error logs didn't give me a lot to go on [21:59:16] error logs didn't give me a lot to go on [21:59:18] your weclome :) [21:59:18] your weclome :) [21:59:35] is there a stable-2.16 upstream branch already? [21:59:35] is there a stable-2.16 upstream branch already? [21:59:36] thcipriani im pretty sure it's not the gerrit upgrade that caused it. [21:59:36] thcipriani im pretty sure it's not the gerrit upgrade that caused it. [21:59:39] thcipriani yes [21:59:39] thcipriani yes [21:59:46] upstream are ready to release 2.16 [21:59:46] upstream are ready to release 2.16 [21:59:47] today [21:59:48] today [22:00:19] I wish I were as certain as you are about the upgrade :) [22:00:19] I wish I were as certain as you are about the upgrade :) [22:00:25] heh [22:00:25] heh [22:00:57] there is a jetty upgrade in 2.15.7 though [22:00:57] there is a jetty upgrade in 2.15.7 though [22:01:02] to fix some CVE [22:01:02] to fix some CVE [22:01:06] anyway, I installed javamelody to hopefully get some more insight into what could be happening. [22:01:06] anyway, I installed javamelody to hopefully get some more insight into what could be happening. [22:01:10] :) [22:01:10] :) [22:01:41] might block some bots from crawling gitiles as well [22:01:41] might block some bots from crawling gitiles as well [22:01:54] yep, +1 to that [22:01:54] yep, +1 to that [22:02:43] thcipriani from what i can tell re 2.16, is it's now full notedb! (still requires a db connection) [22:02:43] thcipriani from what i can tell re 2.16, is it's now full notedb! (still requires a db connection) [22:03:05] heh, dunno if I want that :) [22:03:05] heh, dunno if I want that :) [22:03:10] but won't really be using the db to store anything any more! [22:03:10] but won't really be using the db to store anything any more! [22:03:12] heh [22:03:12] heh [22:03:15] I like databases [22:03:15] I like databases [22:03:16] thcipriani it's only groups [22:03:16] thcipriani it's only groups [22:03:21] heh [22:03:21] heh [22:03:22] yeah [22:03:22] yeah [22:04:27] thcipriani https://gerrit-review.googlesource.com/c/gerrit/+/75801 [22:04:27] thcipriani https://gerrit-review.googlesource.com/c/gerrit/+/75801 [22:04:34] that was the last thing to move to notedb [22:04:34] that was the last thing to move to notedb [22:04:41] so from 3.0+ reviewdb is being killed [22:04:41] so from 3.0+ reviewdb is being killed [22:05:46] thcipriani another thing is now you can choose what gerritbot will post per repo. [22:05:46] thcipriani another thing is now you can choose what gerritbot will post per repo. [22:06:04] its-base was updated to support a actions.conf file in refs/meta/config [22:06:04] its-base was updated to support a actions.conf file in refs/meta/config [22:06:34] that's kinda neat. [22:06:34] that's kinda neat. [22:06:50] yep [22:06:50] yep [22:06:56] i've tested it and it indeed works [22:06:56] i've tested it and it indeed works [22:08:09] thcipriani also 75k of lines was deleted from gerrit master (3.0) [22:08:09] thcipriani also 75k of lines was deleted from gerrit master (3.0) [22:08:20] because of the removal of gwtui (so smaller war sizes too!) [22:08:20] because of the removal of gwtui (so smaller war sizes too!) [22:09:21] ah [22:09:21] ah [22:09:55] deploy to 1 server doesn't take too terribly long with 7xM .war [22:09:55] deploy to 1 server doesn't take too terribly long with 7xM .war [22:10:25] heh [22:10:25] heh [22:10:27] thcipriani: https://gerrit.wikimedia.org/r/monitoring has a donate button o.O [22:10:28] thcipriani: https://gerrit.wikimedia.org/r/monitoring has a donate button o.O [22:10:38] donate? [22:10:38] donate? [22:11:03] https://screenshots.firefox.com/tLHkqX2HlRBzS1tD/gerrit.wikimedia.org [22:11:03] https://screenshots.firefox.com/tLHkqX2HlRBzS1tD/gerrit.wikimedia.org [22:11:16] oh [22:11:16] oh [22:11:28] anyways, just weird [22:11:29] anyways, just weird [22:11:32] sure enough [22:11:32] sure enough [22:11:36] it's not loading any external resources so it's fine [22:11:36] it's not loading any external resources so it's fine [22:12:05] * paladox wonders how that will work in gerrit 3.0 :) [22:12:05] ACTION wonders how that will work in gerrit 3.0 :) [22:12:11] will have to be converted to polygerrit [22:12:11] will have to be converted to polygerrit [22:12:36] seems like the project could use some donations [22:12:36] seems like the project could use some donations [22:13:03] I hadn't noticed it before, but it's there for jenkins as well [22:13:03] I hadn't noticed it before, but it's there for jenkins as well [22:13:58] thcipriani delete-project is being made a core plugin :) [22:13:58] thcipriani delete-project is being made a core plugin :) [22:14:38] i also finally fixed the create change dialog to not show the private button if you have it disabled! [22:14:38] i also finally fixed the create change dialog to not show the private button if you have it disabled! [22:14:39] https://gerrit-review.googlesource.com/c/gerrit/+/204371 [22:14:39] https://gerrit-review.googlesource.com/c/gerrit/+/204371 [22:18:05] thcipriani i have been testing 2.16 on https://gerrit.git.wmflabs.org/r/ anyways :) [22:18:05] thcipriani i have been testing 2.16 on https://gerrit.git.wmflabs.org/r/ anyways :) [22:18:19] i have a few patches in puppet for the 2.16 upgrade! [22:18:19] i have a few patches in puppet for the 2.16 upgrade! [22:18:31] since it needs some soy template updates. [22:18:31] since it needs some soy template updates. [22:18:46] nice :) [22:18:46] nice :) [22:19:10] and also it includes a new logger frontend called flogger [22:19:10] and also it includes a new logger frontend called flogger [22:19:17] that has backwords compat with log4j. [22:19:17] that has backwords compat with log4j. [22:19:32] thcipriani ohhh and another thing there's a new request tracing feature. [22:19:33] thcipriani ohhh and another thing there's a new request tracing feature. [22:20:05] https://www.gerritcodereview.com/2.16.html#request-tracing [22:20:05] https://www.gerritcodereview.com/2.16.html#request-tracing [22:20:07] don't you have a patch re:flogger [22:20:07] don't you have a patch re:flogger [22:20:12] yep [22:20:12] yep [22:20:34] for the backwords compat with log4j (it's done by init) but puppet will just put the old version back. [22:20:34] for the backwords compat with log4j (it's done by init) but puppet will just put the old version back. [22:21:29] https://gerrit-documentation.storage.googleapis.com/Documentation/2.16/user-request-tracing.html [22:21:29] https://gerrit-documentation.storage.googleapis.com/Documentation/2.16/user-request-tracing.html [22:22:00] thcipriani that ^^ may make your life easier (but not sure) [22:22:01] thcipriani that ^^ may make your life easier (but not sure) [22:23:25] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Aklapper) To me that sounds like setting "Priority" to "lowest"? :P [22:23:25] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Aklapper) To me that sounds like setting "Priority" to "lowest"? :P [22:24:27] thcipriani i also wonder if we should open https://gerrit.wikimedia.org/r/monitoring to gerrit managers (if it contains nothing private) [22:24:28] thcipriani i also wonder if we should open https://gerrit.wikimedia.org/r/monitoring to gerrit managers (if it contains nothing private) [22:25:22] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10reosarevok) Fine, we'll try just doing these things for now and come back later if we feel we need anything else :) Feel free to close this for now as whatever seems appropriate. [22:25:22] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10reosarevok) Fine, we'll try just doing these things for now and come back later if we feel we need anything else :) Feel free to close this for now as whatever seems appropriate. [22:27:49] paladox: nothing in the default view, afaict. Usernames. There are buttons to do heap dumps and the like which I'm less certain about. [22:27:49] paladox: nothing in the default view, afaict. Usernames. There are buttons to do heap dumps and the like which I'm less certain about. [22:28:01] hmm [22:28:01] hmm [22:28:52] nothing in the default view *that's private*, I mean. [22:28:52] nothing in the default view *that's private*, I mean. [22:29:11] ah great :) [22:29:11] ah great :) [22:29:43] Also the gerrit manager group is small (and trusted users i presume are the only ones in there) [22:29:43] Also the gerrit manager group is small (and trusted users i presume are the only ones in there) [22:29:51] https://gerrit.wikimedia.org/r/#/admin/groups/119,members [22:29:51] https://gerrit.wikimedia.org/r/#/admin/groups/119,members [22:32:21] (03PS1) 10Paladox: Allow "Gerrit Managers" to view /monitoring [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/473636 [22:32:21] (03PS1) 10Paladox: Allow "Gerrit Managers" to view /monitoring [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/473636 [22:32:25] thcipriani ^^ [22:32:25] thcipriani ^^ [22:34:08] paladox: I mean to say that while nothing in the view is private, some of the capabilities may be. Let me look over what I can make this do for a bit :) [22:34:08] paladox: I mean to say that while nothing in the view is private, some of the capabilities may be. Let me look over what I can make this do for a bit :) [22:34:18] ah ok [22:34:18] ah ok [22:34:51] Im planning to update gerrit.git.wmflabs.org in a few hours to the final 2.16 update [22:34:51] Im planning to update gerrit.git.wmflabs.org in a few hours to the final 2.16 update [22:35:11] which means i can build the javamolody thingy (or what ever it's called :)) [22:35:11] which means i can build the javamolody thingy (or what ever it's called :)) [22:36:19] yeah, it wasn't too bad to build, required an extra library that had to go in [review_site]/lib/ [22:36:19] yeah, it wasn't too bad to build, required an extra library that had to go in [review_site]/lib/ [22:36:38] build instructions in the plugin documentation worked, I just didn't read them initially [22:36:38] build instructions in the plugin documentation worked, I just didn't read them initially [22:36:54] heh [22:36:54] heh [22:39:34] (03CR) 10Hashar: [C: 031] "The JavaMelody interface lets you kill java threads, run the java garbage collector and I have no idea what kind of private informations i" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/473636 (owner: 10Paladox) [22:39:34] (03CR) 10Hashar: [C: 031] "The JavaMelody interface lets you kill java threads, run the java garbage collector and I have no idea what kind of private informations i" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/473636 (owner: 10Paladox) [22:40:31] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 4 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) Now rec... [22:40:31] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-PropertySuggester, and 4 others: PHP 7.2 gate-and-submit for some extensions fails due to deprecated function each() in CirrusSearch / Elastica - https://phabricator.wikimedia.org/T209508 (10Jdforrester-WMF) Now rec... [22:58:42] thcipriani someone else noticed your thread and posted it into the hackathon channel :P [22:58:42] thcipriani someone else noticed your thread and posted it into the hackathon channel :P [22:59:37] thcipriani they are asking weather updating to 2.15.7 may help [22:59:37] thcipriani they are asking weather updating to 2.15.7 may help [22:59:46] as it includes a update to jetty (with bug fixes) [22:59:46] as it includes a update to jetty (with bug fixes) [23:01:13] thcipriani quite possibly our issue is related to https://github.com/eclipse/jetty.project/issues/347? [23:01:13] thcipriani quite possibly our issue is related to https://github.com/eclipse/jetty.project/issues/347? [23:03:28] hmm [23:03:28] hmm [23:03:43] that has it already in 9.3 (the fix) [23:03:43] that has it already in 9.3 (the fix) [23:27:06] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Aklapper) 05Open>03declined Alright, I'll set the status to declined for the time being, but please feel free to set the status back to "Open" via the {nav name=Add Action... > Change Status} dropdown i... [23:27:06] 10Project-Admins: Estonian version of "needs-volunteer" tag - https://phabricator.wikimedia.org/T209354 (10Aklapper) 05Open>03declined Alright, I'll set the status to declined for the time being, but please feel free to set the status back to "Open" via the {nav name=Add Action... > Change Status} dropdown i... [23:36:56] thcipriani fyi there's a rate limiter plugin in gerrit now! [23:36:56] thcipriani fyi there's a rate limiter plugin in gerrit now! [23:36:57] https://gerrit-review.googlesource.com/q/project:plugins%252Frate-limiter [23:36:57] https://gerrit-review.googlesource.com/q/project:plugins%252Frate-limiter [23:57:14] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10Aklapper) Quick'n'dirty'n'unstable [user script](https://en.wikipedia.org/wiki/Userscript): ` if (document.getElementsByClassName('phui-crumb-name').length > 1) { var taskId = docu... [23:57:14] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10Aklapper) Quick'n'dirty'n'unstable [user script](https://en.wikipedia.org/wiki/Userscript): ` if (document.getElementsByClassName('phui-crumb-name').length > 1) { var taskId = docu...