[00:50:35] jdlrobson: Would like to start warning for $.each soon :) - wanna close up on https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/451888/ ? [01:26:12] 10Phabricator: Wrong phabricator patch statistics - https://phabricator.wikimedia.org/T202954 (10IKhitron) [01:55:19] 10Continuous-Integration-Config, 10RelEng-Archive-FY201718-Q2, 10Composer, 10Upstream, 10Wikimedia-log-errors (Shared Build Failure): Error "TransportException 404 Not Found" in Jenkins jobs using composer - https://phabricator.wikimedia.org/T182266 (10Krinkle) >>! In T182266#4259273, @Tgr wrote: > [[htt... [02:18:47] 10Phabricator: Wrong phabricator patch statistics - https://phabricator.wikimedia.org/T202954 (10Zoranzoki21) HI @IKhitron, I checked https://phabricator.wikimedia.org/people/commits/1454/ and I saw to all is related to my patch for enabling extension newsletter on hewiki https://gerrit.wikimedia.org/r/#/c/opera... [02:24:17] 10Phabricator: Wrong phabricator patch statistics - https://phabricator.wikimedia.org/T202954 (10Zoranzoki21) 05Open>03Invalid I removed you as reviewer from patch https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/381537/ and rebased. This not happened again. I think to before 4 months was bug... [04:51:00] PROBLEM - Free space - all mounts on integration-slave-docker-1025 is CRITICAL: CRITICAL: integration.integration-slave-docker-1025.diskspace.root.byte_percentfree (<22.22%) [05:11:06] RECOVERY - Free space - all mounts on integration-slave-docker-1025 is OK: OK: All targets OK [06:28:46] (03CR) 10Tarrow: [C: 031] mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [06:49:29] (03CR) 10Tarrow: "> Patch Set 2:" (0310 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [07:18:02] (03PS1) 10Hashar: Add missing operations/debs repositories [integration/config] - 10https://gerrit.wikimedia.org/r/455768 [07:18:14] (03CR) 10Hashar: [C: 032] Add missing operations/debs repositories [integration/config] - 10https://gerrit.wikimedia.org/r/455768 (owner: 10Hashar) [07:19:40] (03Merged) 10jenkins-bot: Add missing operations/debs repositories [integration/config] - 10https://gerrit.wikimedia.org/r/455768 (owner: 10Hashar) [07:33:27] 10Release-Engineering-Team (Watching / External), 10Core-Platform-Team, 10PoolCounter, 10Patch-For-Review: Fix tests of PoolCounter extension - https://phabricator.wikimedia.org/T178517 (10hashar) You can check my commit message at https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PoolCounter/+/41702... [07:40:28] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Quibble, 10Patch-For-Review, 10User-zeljkofilipin: Quibble does not install ffmpeg - https://phabricator.wikimedia.org/T193157 (10hashar) 05Open>03Resolved I have confirmed ffmpeg is indeed in the containers. Version 3.2.12 for Stretch a... [07:40:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Video recording for Selenium tests in Node.js - https://phabricator.wikimedia.org/T179188 (10hashar) [08:06:51] (03CR) 10Thiemo Kreuz (WMDE): "I checked the implementation of this sniff and find it pretty scary. It triggers like 20 warnings or so. Only one is not auto-fixable, tha" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/455626 (owner: 10Umherirrender) [08:11:26] (03CR) 10Legoktm: "The test run job on our CI is broken (for a long time actually)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/455626 (owner: 10Umherirrender) [08:11:51] I can't quote properly, oops [08:25:38] legoktm: I will ask around for the releng/mediawiki-phpcs-dryrun docker container :] [08:34:20] 08:31:31 + git remote add zuul git://contint2001.wikimedia.org/operations/puppet 08:31:32 + git pull --quiet zuul production 08:31:32 fatal: read error: Connection reset by peer [08:34:24] is this known? [08:34:45] (causes verify -1) [08:35:02] hashar: ^^ [08:35:25] apergos: yup that is not publicly accessible [08:35:37] apergos: ho is that on CI? [08:35:46] https://integration.wikimedia.org/ci/job/operations-puppet-tests-docker/27008/console [08:35:54] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454712/ [08:36:25] unless it was some other issue and I just didn't see it [08:36:54] the git-daemon had an issue of some sort I guess. I will dig in the logs [08:37:01] ok [08:37:32] Aug 28 08:31:32 contint2001 git-daemon[21088]: fatal: packet write with format failed: Connection reset by peer [08:38:01] huh [08:38:11] apergos: I have no idea. I guess ypou can just recheck the change [08:38:16] will do [08:39:25] huh [08:39:37] well that's annoying but at least the workaround is easy, thanks [08:59:28] (03PS26) 10Hashar: Selenium daily tests for beta using Docker/wdio [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) [09:01:31] zeljkof: I have rebased your patch for selenium daily tests with docker/wdio :] [09:02:07] and my bet is that i missed something and it will explode. Will see on https://integration.wikimedia.org/ci/job/selenium-daily-beta-CirrusSearch/1/console [09:03:16] (03PS27) 10Hashar: Selenium daily tests for beta using Docker/wdio [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) [09:03:41] hashar: :D [09:03:43] (03CR) 10Hashar: "Add ZUUL_PIPELINE=postmerge to force saving the cache via Castor." [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [09:04:16] I'll review it, I'm still catching up, I don't feel very good, spending a few days with about 40 kids does not improve health [09:04:22] hashar: ^ [09:04:39] zeljkof: bah [09:04:44] I am running all them [09:20:21] (03CR) 10Hashar: [C: 031] "Works for me. A lot of repositories are missing the 'selenium-daily' script though and thus are failing. I have made the new job to only e" [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [09:42:54] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests: Add contint-roots to releases{1,2}001 - https://phabricator.wikimedia.org/T201470 (10ArielGlenn) If it's not just about installing the package and restarting but also troubleshooting, it sounds like you'll want an SRE pers... [10:16:20] 10Phabricator: Wrong phabricator patch statistics - https://phabricator.wikimedia.org/T202954 (10IKhitron) >>! In T202954#4536928, @Zoranzoki21 wrote: > Closing as invalid, because code related to this task is not edited. Thank you, but I think this page must be emptied. It shows wrong information about me. [10:17:47] PROBLEM - Free space - all mounts on integration-slave-docker-1010 is CRITICAL: CRITICAL: integration.integration-slave-docker-1010.diskspace.root.byte_percentfree (<10.00%) [10:19:03] PROBLEM - Free space - all mounts on integration-slave-docker-1025 is CRITICAL: CRITICAL: integration.integration-slave-docker-1025.diskspace.root.byte_percentfree (<12.50%) [10:27:42] (03PS5) 10Hashar: mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [10:27:44] RECOVERY - Free space - all mounts on integration-slave-docker-1010 is OK: OK: All targets OK [10:28:48] (03PS6) 10Hashar: mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [10:28:51] (03CR) 10Hashar: mysql: add ability to dump before exit (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [10:41:43] (03CR) 10Pablo Grass (WMDE): mysql: add ability to dump before exit (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [11:06:48] (03PS4) 10Pablo Grass (WMDE): readme: how to reproduce a CI build [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) [11:59:46] 10Release-Engineering-Team, 10MediaWiki-extensions-WikimediaIncubator, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Amire80) >>! In T165585#4534589, @StevenJ81 wrote: > Liuxinyu posted it, not me. But Wolverène posted it at the end of the discus... [12:04:07] (03CR) 10Hashar: mysql: add ability to dump before exit (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [12:04:17] (03PS7) 10Hashar: mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [12:06:33] (03PS8) 10Hashar: mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [12:07:57] 10Scap, 10MediaWiki-Vendor, 10Patch-For-Review: Scap sync fails because it tries to lint test files - https://phabricator.wikimedia.org/T202984 (10Tgr) [12:09:56] 10Scap, 10MediaWiki-Vendor, 10Patch-For-Review: Scap sync fails because it tries to lint test files - https://phabricator.wikimedia.org/T202984 (10Tgr) [12:10:37] 10Scap, 10MediaWiki-Vendor, 10Patch-For-Review: Scap sync fails because it tries to lint test files - https://phabricator.wikimedia.org/T202984 (10Tgr) [12:24:29] (03CR) 10Hashar: [C: 031] "Sorry I have messed up a bit. Gave it a try locally (using --skip-all to skip running tests) and that seems to be working properly." [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [12:31:17] 10Phabricator: Phabricator lists incorrect "commits" for user (due to having been a "reviewer" on Gerrit?) - https://phabricator.wikimedia.org/T202954 (10Aklapper) [12:42:24] (03PS2) 10Hashar: phan-seccheck should use a cache [integration/config] - 10https://gerrit.wikimedia.org/r/454515 (https://phabricator.wikimedia.org/T202225) [12:43:02] (03CR) 10Hashar: [C: 032] phan-seccheck should use a cache [integration/config] - 10https://gerrit.wikimedia.org/r/454515 (https://phabricator.wikimedia.org/T202225) (owner: 10Hashar) [12:43:16] 10Scap, 10MediaWiki-Vendor, 10Patch-For-Review: Scap sync fails because it tries to lint test files - https://phabricator.wikimedia.org/T202984 (10Tgr) Solved for now by adding all files from the lint whitelist to gitignore and removing them from the repo. In the longer term, either scap should call composer... [12:43:22] 10Continuous-Integration-Config, 10Patch-For-Review: mwext-php70-phan-seccheck-docker isn't using composer cache - https://phabricator.wikimedia.org/T202225 (10hashar) 05Open>03Resolved a:03hashar Should be good now. [12:45:23] (03Merged) 10jenkins-bot: phan-seccheck should use a cache [integration/config] - 10https://gerrit.wikimedia.org/r/454515 (https://phabricator.wikimedia.org/T202225) (owner: 10Hashar) [13:03:16] (03PS9) 10Pablo Grass (WMDE): mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) [13:07:47] (03CR) 10Anomie: "Terminology looks good now." (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449149 (https://phabricator.wikimedia.org/T171520) (owner: 10PleaseStand) [13:09:41] (03CR) 10Pablo Grass (WMDE): [C: 031] "I tried it and it still does what I expect." (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [13:35:34] PROBLEM - SSH on integration-slave-docker-1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:06] (03PS1) 10Hashar: Move phpcs dryrun to the generic composer image [integration/config] - 10https://gerrit.wikimedia.org/r/455835 (https://phabricator.wikimedia.org/T200722) [13:36:43] (03CR) 10Hashar: "I gave it a try on https://integration.wikimedia.org/ci/job/mw-tools-codesniffer-mwcore-testrun-2/3/console There are a lot of errors and" [integration/config] - 10https://gerrit.wikimedia.org/r/455835 (https://phabricator.wikimedia.org/T200722) (owner: 10Hashar) [13:45:25] RECOVERY - SSH on integration-slave-docker-1005 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u5 (protocol 2.0) [14:25:11] (03PS2) 10Hashar: Move phpcs dryrun to the generic composer image [integration/config] - 10https://gerrit.wikimedia.org/r/455835 (https://phabricator.wikimedia.org/T200722) [14:25:28] (03CR) 10Hashar: "https://integration.wikimedia.org/ci/job/mw-tools-codesniffer-mwcore-testrun-2/9/console :]" [integration/config] - 10https://gerrit.wikimedia.org/r/455835 (https://phabricator.wikimedia.org/T200722) (owner: 10Hashar) [14:53:02] PROBLEM - Free space - all mounts on integration-slave-docker-1001 is CRITICAL: CRITICAL: integration.integration-slave-docker-1001.diskspace.root.byte_percentfree (<50.00%) [15:20:40] (03PS1) 10Hashar: Add support for Zuul --branch and --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455855 [15:31:11] (03PS2) 10Hashar: Add support for Zuul --branch and --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455855 [15:31:39] (03CR) 10Hashar: "Testing it using:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/455855 (owner: 10Hashar) [15:33:00] RECOVERY - Free space - all mounts on integration-slave-docker-1001 is OK: OK: All targets OK [15:35:55] 10Release-Engineering-Team, 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10thcipriani) [15:36:29] 10Release-Engineering-Team, 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10thcipriani) For an example of the problem see T202952 [15:37:16] (03CR) 10Hashar: [C: 032] "> I tried it and it still does what I expect." (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [15:38:01] (03Merged) 10jenkins-bot: mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [15:38:32] (03CR) 10jenkins-bot: mysql: add ability to dump before exit [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [15:43:35] 10Release-Engineering-Team, 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10thcipriani) [15:47:29] (03CR) 10Pablo Grass (WMDE): mysql: add ability to dump before exit (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452384 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [15:51:48] 10Release-Engineering-Team, 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10faidon) Thanks for filing this! I lost about an hour debugging and (re-)fixing the above issue today, so +1 to everything you said :) Let's sort this out and all other potential cases l... [15:52:09] (03PS3) 10Hashar: Add support for Zuul --branch and --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455855 [15:58:03] 10Release-Engineering-Team, 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10mmodell) I agree that we should use the upstream repo from puppet and I should have refactored the puppet code as soon as I created rKEYHOLDER. The question then is, do we install via deb... [15:58:46] (03CR) 10Umherirrender: "I have fixed a found issue with Iae57172138f8887e37ed401b443531259808e678, but after that it passed on mediawiki/core" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/455626 (owner: 10Umherirrender) [16:04:19] (03CR) 10Hashar: [C: 032] "Some progress achieved via:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/455855 (owner: 10Hashar) [16:05:09] (03Merged) 10jenkins-bot: Add support for Zuul --branch and --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455855 (owner: 10Hashar) [16:05:40] (03CR) 10jenkins-bot: Add support for Zuul --branch and --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455855 (owner: 10Hashar) [16:06:26] !log tagged quibble 0.0.22 5093492821693918da4fecdcb15a8ca831ce560f [16:06:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:10:45] (03PS1) 10Hashar: docker: quibble 0.0.22 [integration/config] - 10https://gerrit.wikimedia.org/r/455860 (https://phabricator.wikimedia.org/T192132) [16:11:02] (03CR) 10Hashar: [C: 032] docker: quibble 0.0.22 [integration/config] - 10https://gerrit.wikimedia.org/r/455860 (https://phabricator.wikimedia.org/T192132) (owner: 10Hashar) [16:11:24] 10MediaWiki-Codesniffer: PHPCS should make sure @covers tags are absolute - https://phabricator.wikimedia.org/T183218 (10Umherirrender) That is correct, the trait is not used everywhere. It could be enforced with phpcs when a class extends the PHPUnit\Framework\TestCase class. Than the test itself and all tests... [16:12:43] (03Merged) 10jenkins-bot: docker: quibble 0.0.22 [integration/config] - 10https://gerrit.wikimedia.org/r/455860 (https://phabricator.wikimedia.org/T192132) (owner: 10Hashar) [16:13:08] (03PS1) 10Hashar: Switch Jenkins jobs to Quibble 0.0.22 containers [integration/config] - 10https://gerrit.wikimedia.org/r/455861 (https://phabricator.wikimedia.org/T192132) [16:13:38] !log building docker containers for quibble 0.0.22 [16:13:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:14:54] (03PS2) 10Hashar: Switch Jenkins jobs to Quibble 0.0.22 containers [integration/config] - 10https://gerrit.wikimedia.org/r/455861 (https://phabricator.wikimedia.org/T192132) [16:24:58] marxarelli: here [16:24:59] !sal [16:24:59] https://tools.wmflabs.org/sal/releng [16:25:03] magick! [16:56:49] (03CR) 10Hashar: [C: 032] Switch Jenkins jobs to Quibble 0.0.22 containers [integration/config] - 10https://gerrit.wikimedia.org/r/455861 (https://phabricator.wikimedia.org/T192132) (owner: 10Hashar) [16:58:51] (03Merged) 10jenkins-bot: Switch Jenkins jobs to Quibble 0.0.22 containers [integration/config] - 10https://gerrit.wikimedia.org/r/455861 (https://phabricator.wikimedia.org/T192132) (owner: 10Hashar) [16:59:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Xvfb causes _XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created - https://phabricator.wikimedia.org/T202710 (10hashar) 05Open>03Resolved Released in Quibbl... [16:59:55] !log Switched Jenkins jobs to Quibble 0.0.22 | https://gerrit.wikimedia.org/r/455861 [16:59:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:03:23] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review: Mails through deployment-mx SPF & DKIM fails - https://phabricator.wikimedia.org/T87338 (10Krenair) And actually something I heard the other day made it sound like we may not be able to do that by default with our neutron setup either. [17:07:20] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review: Mails through deployment-mx SPF & DKIM fails - https://phabricator.wikimedia.org/T87338 (10Krenair) 05Open>03Resolved a:03Krenair Anyway although we may struggle to make beta behaviour exactly mirror prod, the initial purpose of this tick... [17:21:06] hasharAway: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/13140/console quibble is throwing exceptions [17:25:03] 10Continuous-Integration-Config, 10Quibble: CI is failing on all open patches - https://phabricator.wikimedia.org/T203009 (10MarcoAurelio) [17:25:13] 10Continuous-Integration-Config, 10Quibble: CI is failing on all open patches - https://phabricator.wikimedia.org/T203009 (10MarcoAurelio) p:05Triage>03Unbreak! [17:26:18] (03CR) 10MarcoAurelio: "This might be causing all patches to fail CI: T203009." [integration/config] - 10https://gerrit.wikimedia.org/r/455861 (https://phabricator.wikimedia.org/T192132) (owner: 10Hashar) [17:26:21] (03PS1) 10Legoktm: Revert "Switch Jenkins jobs to Quibble 0.0.22 containers" [integration/config] - 10https://gerrit.wikimedia.org/r/455872 (https://phabricator.wikimedia.org/T203009) [17:26:43] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches - https://phabricator.wikimedia.org/T203009 (10MarcoAurelio) https://gerrit.wikimedia.org/r/#/c/integration/config/+/455861/ ? [17:26:48] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches - https://phabricator.wikimedia.org/T203009 (10Paladox) [17:29:21] (03CR) 10Legoktm: [C: 032] "Reverted all jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/455872 (https://phabricator.wikimedia.org/T203009) (owner: 10Legoktm) [17:30:22] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10Legoktm) [17:30:36] !log reverted back to previous quibble version (T203009) [17:30:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:30:40] T203009: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 [17:31:30] (03Merged) 10jenkins-bot: Revert "Switch Jenkins jobs to Quibble 0.0.22 containers" [integration/config] - 10https://gerrit.wikimedia.org/r/455872 (https://phabricator.wikimedia.org/T203009) (owner: 10Legoktm) [17:32:18] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10Legoktm) p:05Unbreak!>03High Lowering priority slightly since I reverted the quibble upgrade. [17:34:27] 10Phabricator, 10Pywikibot-core, 10Proposal, 10Pywikibot-RfCs: RFC: Add pywikibot-core to all new pywikibot-script/module/class tasks automatically - https://phabricator.wikimedia.org/T170090 (10Aklapper) >>! In T170090#3830520, @Dvorapa wrote: > * move the current existing projects (ie: #pywikibot-archive... [17:34:31] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10Legoktm) 5093492821693918da4fecdcb15a8ca831ce560f is the problematic commit. [17:41:01] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10hashar) 05Open>03Resolved a:03hashar [17:41:13] legoktm: yeah insufficient testing on my behalf :/// [17:41:26] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10hashar) a:05hashar>03Legoktm [17:46:05] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10MarcoAurelio) Thank you for expeditiously identifying and fixing the issue. [17:53:18] (03PS1) 10Hashar: Fix empty exception on empty --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455877 (https://phabricator.wikimedia.org/T203009) [17:54:06] 10Continuous-Integration-Config, 10Quibble: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10hashar) 05Resolved>03Open It is all my fault, I have not properly tested it :( [17:54:42] such a pity [17:58:33] Project beta-scap-eqiad build #220492: 15ABORTED in 7 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/220492/ [17:58:43] ^ me [17:58:44] (03CR) 10Hashar: [C: 032] Fix empty exception on empty --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455877 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [17:59:29] (03Merged) 10jenkins-bot: Fix empty exception on empty --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455877 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [18:00:00] (03CR) 10jenkins-bot: Fix empty exception on empty --project-branch [integration/quibble] - 10https://gerrit.wikimedia.org/r/455877 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [18:00:05] !git log tag Quibble 0.0.23 10035e33c12be68d7940059568f47a1da176ed65 [18:02:22] (03PS1) 10Hashar: docker: quibble 0.0.23 [integration/config] - 10https://gerrit.wikimedia.org/r/455880 (https://phabricator.wikimedia.org/T203009) [18:02:35] (03CR) 10Hashar: [C: 032] docker: quibble 0.0.23 [integration/config] - 10https://gerrit.wikimedia.org/r/455880 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [18:04:17] (03Merged) 10jenkins-bot: docker: quibble 0.0.23 [integration/config] - 10https://gerrit.wikimedia.org/r/455880 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [18:04:20] (03PS1) 10Hashar: Switch Jenkins jobs to Quibble 0.0.23 containers [integration/config] - 10https://gerrit.wikimedia.org/r/455881 (https://phabricator.wikimedia.org/T203009) [18:04:43] !log Rebuilding Docker quibble containers to use Quibble 0.0.23 | T203009 [18:04:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:04:47] T203009: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 [18:05:09] legoktm: thank you :]]]]] [18:05:32] (03CR) 10Hashar: "containers are being build on contint1001 ..." [integration/config] - 10https://gerrit.wikimedia.org/r/455881 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [18:17:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T191065 (10dduvall) a:03dduvall [18:23:54] 10Continuous-Integration-Config, 10PhabTaskGraph: PhabTaskGraph extension cannot find phutil_library - https://phabricator.wikimedia.org/T202515 (10Krinkle) A simpler solution could be a git submodule. Avoids the on-going cost of owning and maintaining a package, whilst not having to duplicate files in Git, an... [18:55:13] !log Bump Jenkins Quibble jobs to 0.0.23 | https://gerrit.wikimedia.org/r/#/c/integration/config/+/455881/ | T203009 [18:55:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:55:18] T203009: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 [18:58:20] 10Continuous-Integration-Config, 10Quibble, 10Patch-For-Review: CI is failing on all open patches due to Quibble 0.0.22 - https://phabricator.wikimedia.org/T203009 (10hashar) 05Open>03Resolved [18:58:32] (03CR) 10Hashar: [C: 032] Switch Jenkins jobs to Quibble 0.0.23 containers [integration/config] - 10https://gerrit.wikimedia.org/r/455881 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [19:02:30] (03Merged) 10jenkins-bot: Switch Jenkins jobs to Quibble 0.0.23 containers [integration/config] - 10https://gerrit.wikimedia.org/r/455881 (https://phabricator.wikimedia.org/T203009) (owner: 10Hashar) [19:06:28] 10Phabricator: Burnup rate is timing out - https://phabricator.wikimedia.org/T203018 (10MGChecker) [19:07:06] 10Phabricator: Burnup rate is timing out - https://phabricator.wikimedia.org/T203018 (10MGChecker) [19:22:01] Project beta-update-databases-eqiad build #27911: 04FAILURE in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/27911/ [19:28:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T191065 (10dduvall) Branch 1.32.0-wmf.19 has been deployed to group0. So far, there's nothing conspicuous in the logs, only {T202674} which @thcipriani says he noticed du... [19:33:50] 10Phabricator: /maniphest/report/burn/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T203018 (10Aklapper) [19:34:01] 10Phabricator: /maniphest/report/burn/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T203018 (10Aklapper) True. Basically same as T125357, just another report (`burn` instead of `project`) [19:41:17] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Jenkins, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 (10Krenair) >>! In T192561#4530411, @Dzahn wrote: > Production deployment servers don't have a... [20:21:56] Project beta-update-databases-eqiad build #27912: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/27912/ [20:23:53] 10Phabricator, 10Pywikibot-core, 10Proposal, 10Pywikibot-RfCs: RFC: Add pywikibot-core to all new pywikibot-script/module/class tasks automatically - https://phabricator.wikimedia.org/T170090 (10Dvorapa) >>! In T170090#4539200, @Aklapper wrote: >>>! In T170090#3830520, @Dvorapa wrote: >> * move the current... [20:24:32] 20:21:56 Exception: ('command: ', "echo 'loginwiki'; /usr/local/bin/mwscript update.php --wiki=loginwiki --quick", 'output: ', "loginwiki\n#!/usr/bin/env php\n[Tue Aug 28 20:21:45 2018] [hphp] [28294:7ff0ccc8f3c0:0:000001] [] \nFatal error: Uncaught exception 'ExtensionDependencyError' with message 'TemplateWizard requires TemplateData to be installed.\n' in /srv/mediawiki-staging/php-master/includes/registration/ExtensionRegistry.php:279\ [20:24:32] nStack trace:\n#0 /srv/mediawiki-staging/php-master/includes/registration/ExtensionRegistry.php(159): ExtensionRegistry->readFromQueue()\n#1 /srv/mediawiki-staging/php-master/includes/Setup.php(122): ExtensionRegistry->loadFromQueue()\n#2 /srv/mediawiki-staging/php-master/maintenance/doMaintenance.php(81): include()\n#3 /srv/mediawiki-staging/php-master/maintenance/update.php(245): include()\n#4 /srv/mediawiki-staging/multiversion/MWScript. [20:24:32] php(100): include()\n#5 {main}\n") [20:26:41] Uhhhhh. [20:26:45] James_F: :( [20:27:30] legoktm: Eurgh. wmf.18 loginwiki but wmf.19 group0 wikis, I assume? [20:27:40] legoktm: Wait, TemplateWizard isn't enabled on loginwiki or any prod wikis? [20:28:18] https://login.wikimedia.beta.wmflabs.org/ [20:28:48] Hmm. [20:29:00] That shouldn't complain? [20:30:01] see https://gerrit.wikimedia.org/r/456015 [20:30:10] most non-essential extension are disabled on loginwiki [20:30:21] Ah, yeah. [20:36:24] Project beta-scap-eqiad build #220500: 04FAILURE in 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/220500/ [20:36:55] well that's new [20:37:24] Permission denied? [20:39:23] that has always been some inadvertly chmod [20:39:34] * thcipriani checks it out [20:39:40] check chmod for the files and see if jenkins-deploy is allowed [20:42:03] hi thifranc [20:42:16] so.. i was just showing a brand new user around and told him to get a wikitech user and then use it to login on Gerrit and Phabricator, that is thifranc [20:42:27] i can see his user in LDAP and he can login on Gerrit [20:42:43] but he cannot login on Phabricator. using the same credentials [20:42:57] now i suspect it might be antivandalism [20:43:39] thifranc: oh, look at this: https://phabricator.wikimedia.org/p/Thifranc/ [20:43:47] LDAP User [20:43:48] Unknown [20:44:06] i think you already linked your Mediawiki user to Phabricator [20:44:28] question to others; can he change that and use the wikitech user? [20:44:35] you can add both [20:44:55] https://phabricator.wikimedia.org/settings/user/USERNAME/page/external/ [20:45:13] i really wish phab didn't put the username in the settings url [20:45:42] thifranc: try that ^. first login with your global wiki user and then https://phabricator.wikimedia.org/settings/user/thifranc/page/external/ [20:46:22] so what should work is the user you use on Wikipedia [20:46:22] Awesome, done [20:46:26] :) [21:03:01] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Jenkins, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 (10Dzahn) Thanks, i see: ``` # Parsoid JavaScript dependencies are updated on beta via npm p... [21:05:27] https://www.mediawiki.org/wiki/Wikimedia_Site_Reliability_Engineering talks about "maintaining our presence in 8 locations across the world" [21:05:39] 8 is a bit much. 5 seems more accurate [21:05:44] 10Phabricator, 10Release-Engineering-Team (Kanban): Phabricator DB query request: list repositories and assigned callsigns - https://phabricator.wikimedia.org/T202222 (10mmodell) {P7490} [21:06:43] Is it including eqord and stuff? [21:08:30] maybe that page was written by some marketing people :) [21:08:35] there'll be the 5 numbered datacentres along with knams, eqord, eqdfw, etc. [21:09:11] true, it could be counting eqord [21:09:17] and that esams counts as 2 [21:09:27] knams and esams [21:09:58] Mark added it: https://www.mediawiki.org/w/index.php?title=Wikimedia_Site_Reliability_Engineering&diff=prev&oldid=2752451&diffmode=source [21:13:24] Yippee, build fixed! [21:13:24] Project beta-scap-eqiad build #220501: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/220501/ [21:15:52] 10Phabricator, 10Release-Engineering-Team (Kanban): Phabricator DB query request: list repositories and assigned callsigns - https://phabricator.wikimedia.org/T202222 (10mmodell) See the comments on P7490 for details of the pattern I used with regex search & replace to format the output. [21:16:20] 10Phabricator, 10Release-Engineering-Team (Kanban): Phabricator DB query request: list repositories and assigned callsigns - https://phabricator.wikimedia.org/T202222 (10mmodell) 05Open>03Resolved [21:22:40] Yippee, build fixed! [21:22:41] Project beta-update-databases-eqiad build #27913: 09FIXED in 2 min 39 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/27913/ [21:29:26] (03PS1) 10Hashar: Adapt QA report job xunit reporting [integration/config] - 10https://gerrit.wikimedia.org/r/456025 [21:48:21] (03Abandoned) 10Hashar: Adapt QA report job xunit reporting [integration/config] - 10https://gerrit.wikimedia.org/r/456025 (owner: 10Hashar) [21:57:47] 10Release-Engineering-Team, 10Analytics: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Nuria) [21:58:32] 10Release-Engineering-Team, 10Analytics: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Nuria) Ping @greg et team, while we edit those pages we have not changed them in a while so probably somthing is a miss with the latest wikipedia deployment that has broken them? [21:58:51] 10Release-Engineering-Team, 10Analytics: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Krenair) PHP fatal error: Call to undefined method JsonConfig\JCContent::getLicenseObject() [22:01:28] 10Release-Engineering-Team, 10Analytics, 10MediaWiki-extensions-JsonConfig: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10greg) p:05Triage>03High [22:01:35] 10Release-Engineering-Team, 10Analytics, 10MediaWiki-extensions-JsonConfig: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Jdforrester-WMF) Same bug as {T203006}. :-( [22:02:37] 10Release-Engineering-Team, 10Analytics, 10MediaWiki-extensions-JsonConfig: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Krenair) https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/JsonConfig/+/450397/ is looking fishy [22:17:34] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests: Add contint-roots to releases{1,2}001 - https://phabricator.wikimedia.org/T201470 (10thcipriani) >>! In T201470#4537642, @ArielGlenn wrote: > If it's not just about installing the package and restarting but also troublesho... [22:39:13] 10Release-Engineering-Team, 10Analytics, 10MediaWiki-extensions-JsonConfig, 10Patch-For-Review: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Jdforrester-WMF) Unfortunately it seems my optimism that I50250b483b would fix this was mis-placed; I can't see why the Das... [22:44:07] Project beta-scap-eqiad build #220507: 15ABORTED in 6 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/220507/ [22:44:50] ^ me again [23:02:13] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests: Add contint-roots to releases{1,2}001 - https://phabricator.wikimedia.org/T201470 (10thcipriani) If there are security concerns about adding contint-roots to the releases machines, it might be a better option for SRE to ha... [23:16:10] Hey all, looks like the version tool is broken again? https://tools.wmflabs.org/versions/ [23:16:20] It says that group0 is on wmf.18, but testwiki is on wmf.19 [23:19:21] I bet that's https://phabricator.wikimedia.org/T202734 [23:20:00] I've noticed that happening more frequently recently [23:20:05] not sure what's changed [23:22:50] maybe sync-wikiversions should purge the varnish cache of noc.wm.o? :p [23:30:21] well. it could. [23:31:05] 10Release-Engineering-Team, 10Analytics, 10MediaWiki-extensions-JsonConfig, 10MW-1.32-release-notes (WMF-deploy-2018-08-28 (1.32.0-wmf.19)), and 2 others: Config:Dashiki:* on Meta can't be opened - https://phabricator.wikimedia.org/T203029 (10Krinkle) [23:34:03] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:54] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47522 bytes in 0.819 second response time [23:56:55] thcipriani: we had a similar bug with doc.wm.o being cached for too long, and ended up modifying Apache to send Cache-Control headers for 1h, maybe something similar could be done? [23:57:11] And if we expect /versions/ to truly be 100% up to date, it should also pass a cache busting parameter