[00:14:24] Yippee, build fixed! [00:14:25] Project beta-scap-eqiad build #240162: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240162/ [00:36:43] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Documentation: Improve documentation on Docker-based development environments for new developers - https://phabricator.wikimedia.org/T217614 (10brennen) [00:39:54] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Quibble (marble): Quibble should clone repositories in parallel - https://phabricator.wikimedia.org/T211701 (10greg) a:03hashar [00:40:57] 10Release-Engineering-Team (Backlog), 10Quibble (marble): Quibble should cache node_modules and use npm prune - https://phabricator.wikimedia.org/T211705 (10greg) [00:41:01] 10Release-Engineering-Team (Backlog), 10Quibble (marble): CI Quibble jobs should be feed with git repos and dependencies - https://phabricator.wikimedia.org/T211704 (10greg) [00:41:03] 10Release-Engineering-Team (Backlog), 10Quibble (marble): CI should run composer test/npm test BEFORE running Quibble - https://phabricator.wikimedia.org/T211703 (10greg) [00:41:09] 10Release-Engineering-Team (Backlog), 10Quibble (marble): Quibble initialize step should only clone the target repository - https://phabricator.wikimedia.org/T211702 (10greg) [00:41:27] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Pywikibot, 10Pywikibot-tests: Perform full test suite using Wikimedia CI - https://phabricator.wikimedia.org/T132138 (10greg) [00:41:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Release Pipeline: Pipeline should clone source to a subdirectory of the Jenkins workspace - https://phabricator.wikimedia.org/T216274 (10greg) [00:41:37] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), and 2 others: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903 (10greg) [00:41:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Release Pipeline: The pipeline should provide a way to save artifacts from a stage - https://phabricator.wikimedia.org/T216272 (10greg) [00:41:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Don't hardcode castor url in castor docker container - https://phabricator.wikimedia.org/T216244 (10greg) [00:41:52] 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10greg) [00:41:57] 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Remove UserMerge extension entry from developers/maintainers page - https://phabricator.wikimedia.org/T216091 (10greg) [00:42:04] 10Release-Engineering-Team (Backlog), 10Release Pipeline: Generate per-repo pipeline jobs - https://phabricator.wikimedia.org/T215320 (10greg) [00:42:08] 10Release-Engineering-Team (Backlog), 10serviceops, 10Release Pipeline (Blubber): Base Blubber policy file for CI - https://phabricator.wikimedia.org/T215319 (10greg) [00:42:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10phpunit-patch-coverage, 10Jenkins: Inline styles for patch-coverage HTML artefact blocked by CSP on integration.wikimedia.org - https://phabricator.wikimedia.org/T215115 (10greg) [00:42:33] 10Gerrit, 10Release-Engineering-Team (Backlog): Gerrit: reviewers-by-blame plugin improvements - https://phabricator.wikimedia.org/T214397 (10greg) [00:42:43] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell: Publish the source for phabricator-antivandalism - https://phabricator.wikimedia.org/T202080 (10greg) [00:42:57] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10greg) [00:43:07] 10Phabricator, 10Release-Engineering-Team (Backlog), 10Availability, 10User-MModell, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928 (10greg) [00:43:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Store CI builds output outside Jenkins (e.g. static storage) - https://phabricator.wikimedia.org/T53447 (10greg) [00:43:17] 10Phabricator, 10Release-Engineering-Team (Backlog), 10DBA, 10serviceops, and 2 others: Improve privilege separation for phabricator's config files and mysql credentials - https://phabricator.wikimedia.org/T146055 (10greg) [01:01:26] Project beta-scap-eqiad build #240167: 04FAILURE in 2.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240167/ [01:14:40] Yippee, build fixed! [01:14:40] Project beta-scap-eqiad build #240168: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240168/ [01:16:02] Project beta-scap-eqiad build #240169: 04FAILURE in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240169/ [01:31:29] (03CR) 10Krinkle: "@Hashar this is primarily used for npm-test jobs on non-mediawiki repositories. E.g. visualeditor, oojs, unicode, various performance/* re" [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [01:34:34] Yippee, build fixed! [01:34:35] Project beta-scap-eqiad build #240170: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240170/ [01:35:59] Project beta-scap-eqiad build #240171: 04FAILURE in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240171/ [01:39:33] (03CR) 10Krinkle: "If the are non-quibble-based selenium WDIO tests and they also need to record a video, then I suppose we can install Xvfb still. That woul" [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [01:53:23] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10User-zeljkofilipin: Report results from SonarCloud to Gerrit - https://phabricator.wikimedia.org/T217008 (10kostajh) That seems OK as long as the URL includes the branch (Gerrit patchset number) so that clicking the link takes you to a relevant page o... [02:01:11] Yippee, build fixed! [02:01:11] Project beta-scap-eqiad build #240172: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240172/ [03:01:50] Project mediawiki-core-code-coverage-docker build #4105: 04FAILURE in 1 min 49 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4105/ [04:14:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Pywikibot, 10Pywikibot-tests: Perform full test suite using Wikimedia CI - https://phabricator.wikimedia.org/T132138 (10Xqt) a:05Magul→03None [04:20:40] Project beta-update-databases-eqiad build #32190: 04FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32190/ [04:45:36] Project beta-scap-eqiad build #240187: 04FAILURE in 3.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240187/ [05:04:22] Yippee, build fixed! [05:04:22] Project beta-scap-eqiad build #240188: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240188/ [05:21:12] Yippee, build fixed! [05:21:13] Project beta-update-databases-eqiad build #32191: 09FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32191/ [07:07:05] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10Nik... [07:34:31] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10Nik... [08:10:39] Project beta-scap-eqiad build #240205: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240205/ [08:24:26] Yippee, build fixed! [08:24:27] Project beta-scap-eqiad build #240206: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240206/ [08:27:03] 10Beta-Cluster-Infrastructure, 10Performance-Team: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/494258 [08:28:12] (03Abandoned) 10Jforrester: Add an npm6 variant of the -deploy- image so we can move services over [integration/config] - 10https://gerrit.wikimedia.org/r/478759 (owner: 10Jforrester) [08:28:19] (03Abandoned) 10Jforrester: Docker: Create npm6-node-6-docker and npm6-docker [integration/config] - 10https://gerrit.wikimedia.org/r/478760 (owner: 10Jforrester) [08:41:47] 10Beta-Cluster-Infrastructure, 10Performance-Team, 10Patch-For-Review: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 (10Dzahn) [08:42:07] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit stat... - https://phabricator.wikimedia.org/T216689 [08:59:44] Project beta-code-update-eqiad build #237548: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/237548/ [09:02:46] (03CR) 10Hashar: Revert "Get rid of .gitreview" (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/494068 (https://phabricator.wikimedia.org/T191373) (owner: 10Hashar) [09:02:56] (03PS3) 10Hashar: Revert "Get rid of .gitreview" [tools/scap] - 10https://gerrit.wikimedia.org/r/494068 (https://phabricator.wikimedia.org/T191373) [09:03:27] good morning [09:04:23] Yippee, build fixed! [09:04:23] Project beta-code-update-eqiad build #237549: 09FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/237549/ [09:04:24] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/494068 (https://phabricator.wikimedia.org/T191373) (owner: 10Hashar) [09:05:21] (03CR) 10Dzahn: "there is a circular link somehow. following the "this reverts commit .." link leads back here" [tools/scap] - 10https://gerrit.wikimedia.org/r/494068 (https://phabricator.wikimedia.org/T191373) (owner: 10Hashar) [09:06:45] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Disable jgit gc on gerrit - https://phabricator.wikimedia.org/T217497 (10Dzahn) deployed and service restarted. should be disabled now. [09:07:00] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Disable jgit gc on gerrit - https://phabricator.wikimedia.org/T217497 (10Dzahn) 05Open→03Resolved a:03Dzahn [09:09:22] (03CR) 10Hashar: [C: 04-1] "And scap still has a requirement upon conftool and python-conftool:" [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto) [09:15:01] (03CR) 10Hashar: [C: 03+2] Add CI for new performance/asoranking repo [integration/config] - 10https://gerrit.wikimedia.org/r/494259 (https://phabricator.wikimedia.org/T209857) (owner: 10Gilles) [09:16:30] (03Merged) 10jenkins-bot: Add CI for new performance/asoranking repo [integration/config] - 10https://gerrit.wikimedia.org/r/494259 (https://phabricator.wikimedia.org/T209857) (owner: 10Gilles) [09:38:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Pywikibot, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208 (10Dvorapa) >>! In T186208#4999908, @hashar wrote: > The Wikimedia CI uses a Docker container that runs tox. It recently... [09:44:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Pywikibot, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208 (10Dvorapa) [09:45:22] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Pywikibot, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208 (10Dvorapa) [09:54:42] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Pywikibot, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208 (10Dvorapa) Other tests we are running: | When | What | Where | status | | on every patchset and on +2 | [[https://integr... [10:00:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): cumin fails on integration-cumin: Permission denied (publickey). - https://phabricator.wikimedia.org/T217642 (10hashar) [10:01:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): cumin fails on integration-cumin: Permission denied (publickey). - https://phabricator.wikimedia.org/T217642 (10hashar) On the target instance: ` Mar 5 10:00:01 integration-slave-docker-1021 sshd[14730]: Connection from 172.16.1.103... [10:01:31] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find more CI candidates, divide list based on very hard requirements - https://phabricator.wikimedia.org/T217593 (10zeljkofilipin) a:03zeljkofilipin [10:02:17] 10Release-Engineering-Team (Kanban): Implement "blubber building" CI task for GitLab-CI - https://phabricator.wikimedia.org/T217594 (10zeljkofilipin) [10:02:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): cumin fails on integration-cumin: Permission denied (publickey). - https://phabricator.wikimedia.org/T217642 (10hashar) ` name=/etc/ssh/userkeys/root.d/cumin from="10.68.18.238",no-agent-forwarding,no-port-forwarding,no-x11-forwardin... [10:02:36] 10Release-Engineering-Team (Kanban): Investigate Concourse-CI - https://phabricator.wikimedia.org/T217595 (10zeljkofilipin) [10:08:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): cumin fails on integration-cumin: Permission denied (publickey). - https://phabricator.wikimedia.org/T217642 (10hashar) 05Open→03Resolved That comes from the [[ https://horizon.wikimedia.org/project/puppet/ | project puppet confi... [10:10:08] hashar: there is another public key denied issue: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Babel/+/494067 [10:13:02] !log integration: fixed erroneous ssh key restriction for cumin | T217642 [10:13:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:13:05] T217642: cumin fails on integration-cumin: Permission denied (publickey). - https://phabricator.wikimedia.org/T217642 [10:13:14] Nikerabbit: that is unrealted :) [10:13:31] Nikerabbit: cumin is not used by the CI jobs. It is merely a tool to easily run commands on all the instances [10:13:35] but the jobs do not use cumin [10:16:01] error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup returned status 1 [10:16:07] Failed publickey for jenkins-deploy from 172.16.5.161 port 60110 ssh2: RSA cb:72:8b:fb:08:91:73:aa:12:0a:de:8c:3a:49:f3:28 [10:16:20] Nikerabbit: yeah that is a failure due to the WMCS ldap server [10:17:03] hashar: okay, I wasn't sure if it was related or not [10:18:47] Nikerabbit: best thing to do for those issue is to fill a task anyway ;)) [10:18:51] but yeah [10:18:54] there is not much you can do [10:19:47] hashar: feels like I could spend whole days filing tasks ;) :( [10:21:40] do it ! :] [10:21:53] it scales well and offers a record for later [10:40:53] (03CR) 10Zfilipin: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [10:54:54] (03PS1) 10Lucas Werkmeister (WMDE): Test MassMessage with Flow [integration/config] - 10https://gerrit.wikimedia.org/r/494466 [10:55:36] (03CR) 10Lucas Werkmeister (WMDE): "I’m not sure how to check that the tests actually pass at the moment… if I upload a no-op change to MassMessage that Depends-On this one, " [integration/config] - 10https://gerrit.wikimedia.org/r/494466 (owner: 10Lucas Werkmeister (WMDE)) [11:00:36] (03CR) 10Reedy: "Nope" [integration/config] - 10https://gerrit.wikimedia.org/r/494466 (owner: 10Lucas Werkmeister (WMDE)) [11:12:54] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find more CI candidates, divide list based on very hard requirements - https://phabricator.wikimedia.org/T217593 (10zeljkofilipin) p:05Triage→03Normal [11:52:11] (03CR) 10Krinkle: "@Zfilipin I'm aware that the Mediawiki WDIO jobs collect videos as artefacts, but those run from quibble jobs. This is a different image, " [integration/config] - 10https://gerrit.wikimedia.org/r/494364 (https://phabricator.wikimedia.org/T217545) (owner: 10Hashar) [12:00:32] Project beta-scap-eqiad build #240224: 04FAILURE in 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240224/ [12:04:26] Project beta-scap-eqiad build #240225: 04STILL FAILING in 8.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240225/ [12:05:26] Project beta-scap-eqiad build #240226: 04STILL FAILING in 5.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240226/ [12:14:24] Project beta-scap-eqiad build #240227: 04STILL FAILING in 4.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240227/ [12:20:34] Project beta-update-databases-eqiad build #32198: 04FAILURE in 34 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32198/ [12:24:31] Project beta-scap-eqiad build #240228: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240228/ [12:34:28] Project beta-scap-eqiad build #240229: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240229/ [12:37:19] Project beta-scap-eqiad build #240230: 04STILL FAILING in 3.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240230/ [12:42:43] ooof [12:53:36] Project beta-scap-eqiad build #240231: 04STILL FAILING in 9 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240231/ [12:55:10] re [12:56:32] (03PS5) 10Effie Mouzeli: WIP: Add tests on main.py, add --canary-wait-time [tools/scap] - 10https://gerrit.wikimedia.org/r/493699 [12:56:50] (03CR) 10jerkins-bot: [V: 04-1] WIP: Add tests on main.py, add --canary-wait-time [tools/scap] - 10https://gerrit.wikimedia.org/r/493699 (owner: 10Effie Mouzeli) [13:03:01] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap in job beta-scap-eqiad has random failures: Call to mwscript eval.php returned: None - https://phabricator.wikimedia.org/T217587 (10thcipriani) I'm not sure why it's showing `None` but the actual problem is: ` [thcipriani@deplo... [13:03:26] Project beta-scap-eqiad build #240232: 04STILL FAILING in 8 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240232/ [13:04:41] 10Gerrit, 10Operations: Intermittent slowness on gerrit - https://phabricator.wikimedia.org/T217457 (10jbond) 05Open→03Resolved a:03jbond paladox confirmed via IRC this can be closed [13:04:48] Project beta-scap-eqiad build #240233: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240233/ [13:07:03] 10Gerrit, 10Operations: Intermittent slowness on gerrit - https://phabricator.wikimedia.org/T217457 (10jcrespo) @Mathew.onipe maybe relevant to you ^ [13:08:01] (03CR) 10Giuseppe Lavagetto: WIP: Add tests on main.py, add --canary-wait-time (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/493699 (owner: 10Effie Mouzeli) [13:11:34] 10Gerrit, 10Operations: Intermittent slowness on gerrit - https://phabricator.wikimedia.org/T217457 (10Mathew.onipe) @jcrespo probably... but its all fine now. Thanks! [13:14:24] Project beta-scap-eqiad build #240234: 04STILL FAILING in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240234/ [13:15:54] Project beta-scap-eqiad build #240235: 04STILL FAILING in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240235/ [13:17:12] Call to mwscript eval.php returned: None [13:17:13] bah [13:21:07] Yippee, build fixed! [13:21:07] Project beta-update-databases-eqiad build #32199: 09FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32199/ [13:24:24] Project beta-scap-eqiad build #240236: 04STILL FAILING in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240236/ [13:34:30] Project beta-scap-eqiad build #240237: 04STILL FAILING in 3.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240237/ [13:37:08] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10zeljkofilipin) p:05High→03Low [13:40:01] 10Continuous-Integration-Infrastructure: Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10Nikerabbit) [13:40:59] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find more CI candidates, divide list based on very hard requirements - https://phabricator.wikimedia.org/T217593 (10zeljkofilipin) [13:42:39] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit stat... - https://phabricator.wikimedia.org/T216689 [13:43:40] 10Continuous-Integration-Infrastructure: Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10Nikerabbit) "In most cases, ERROR 1114 results from lack of disk space." [13:44:18] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find more CI candidates, divide list based on very hard requirements - https://phabricator.wikimedia.org/T217593 (10zeljkofilipin) [13:44:27] Project beta-scap-eqiad build #240238: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240238/ [13:54:29] Project beta-scap-eqiad build #240239: 04STILL FAILING in 4.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240239/ [14:04:26] Project beta-scap-eqiad build #240240: 04STILL FAILING in 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240240/ [14:10:32] Project beta-scap-eqiad build #240241: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240241/ [14:14:23] Project beta-scap-eqiad build #240242: 04STILL FAILING in 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240242/ [14:20:39] Project beta-update-databases-eqiad build #32200: 04FAILURE in 39 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32200/ [14:24:28] Project beta-scap-eqiad build #240243: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240243/ [14:28:27] 10Gerrit, 10Release-Engineering-Team: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Dzahn) p:05Triage→03Normal [14:29:09] 10Gerrit, 10Release-Engineering-Team: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Dzahn) Thanks Paladox, this is certainly interesting. [14:30:29] I'm having trouble building a new image with docker-pkg. I've modified Dockerfile.template and added a new entry to the changelog. When I run `docker-pkg --debug -c dockerfiles/config.yaml dockerfiles/java8-sonar-scanner/` I get `ERROR - Unexpected error building image docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.3.0: Image docker-registry.wikimedia.org/releng/java8 not found (image.py:288)` [14:31:26] If I check `docker images` or re-run `docker pull docker-registry.wikimedia.org/releng/java8`, I see that I have the image locally already [14:32:47] 10Release-Engineering-Team (Kanban), 10Scap, 10serviceops, 10Patch-For-Review: Scap: server_groups regression - https://phabricator.wikimedia.org/T217597 (10Joe) [14:32:50] 10Release-Engineering-Team (Kanban), 10Scap, 10serviceops: Deploy scap 3.9.2-1 - https://phabricator.wikimedia.org/T217611 (10Joe) 05Open→03Resolved [14:34:27] Project beta-scap-eqiad build #240244: 04STILL FAILING in 8.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240244/ [14:44:22] Project beta-scap-eqiad build #240245: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240245/ [14:54:22] Project beta-scap-eqiad build #240246: 04STILL FAILING in 4.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240246/ [15:04:21] Project beta-scap-eqiad build #240247: 04STILL FAILING in 4.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240247/ [15:05:10] 10Continuous-Integration-Config, 10Operations, 10Patch-For-Review, 10Puppet: Improve CI checks to cover more of the code base - https://phabricator.wikimedia.org/T215275 (10jbond) 05Open→03Resolved [15:14:27] Project beta-scap-eqiad build #240248: 04STILL FAILING in 9.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240248/ [15:21:11] Yippee, build fixed! [15:21:11] Project beta-update-databases-eqiad build #32201: 09FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32201/ [15:21:27] (03CR) 10Effie Mouzeli: [C: 04-1] "This needs to be two patches, do not merge" [tools/scap] - 10https://gerrit.wikimedia.org/r/493699 (owner: 10Effie Mouzeli) [15:24:28] Project beta-scap-eqiad build #240249: 04STILL FAILING in 9.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240249/ [15:34:27] Project beta-scap-eqiad build #240250: 04STILL FAILING in 9.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240250/ [15:44:28] Project beta-scap-eqiad build #240251: 04STILL FAILING in 9.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240251/ [15:49:32] Project beta-scap-eqiad build #240252: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240252/ [15:54:07] 10Release-Engineering-Team (Kanban), 10Scap, 10serviceops, 10Patch-For-Review: Scap: server_groups regression - https://phabricator.wikimedia.org/T217597 (10thcipriani) 05Open→03Resolved Now that 3.9.2-1 is everywhere, this should be resolved. [15:54:27] Project beta-scap-eqiad build #240253: 04STILL FAILING in 9.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240253/ [16:02:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap in job beta-scap-eqiad has random failures: Call to mwscript eval.php returned: None - https://phabricator.wikimedia.org/T217587 (10hashar) beta-scap-eqiad constantly fails now. The last good build was today March 5th at 11:43:0... [16:04:27] Project beta-scap-eqiad build #240254: 04STILL FAILING in 9.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240254/ [16:14:27] Project beta-scap-eqiad build #240255: 04STILL FAILING in 9.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240255/ [16:24:22] Project beta-scap-eqiad build #240256: 04STILL FAILING in 4.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240256/ [16:32:24] 10Gerrit, 10Release-Engineering-Team: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Paladox) a:03Paladox Im going to work on testing this. [16:34:23] Project beta-scap-eqiad build #240257: 04STILL FAILING in 4.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240257/ [16:41:31] 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10RazeSoldier) What is blocking the release? [16:43:49] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find out what Netsurf does for OS X builds - https://phabricator.wikimedia.org/T217674 (10LarsWirzenius) [16:44:27] Project beta-scap-eqiad build #240258: 04STILL FAILING in 9.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240258/ [16:45:10] 10Release-Engineering-Team (Kanban): Find out about OS X / iOS building/testing from friend - https://phabricator.wikimedia.org/T217675 (10LarsWirzenius) [16:53:42] 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Aklapper) Read the previous comment again. [17:02:31] thcipriani: our zuul embeds the python module 0.7.0 [17:02:49] it comes with some ssl configuration cert/key/ca :) [17:03:50] the the Jenkins Gearman plugin does not have configuration for that ;/ [17:04:40] off [17:05:01] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10Jdforrester-WMF) [17:07:46] Yippee, build fixed! [17:07:46] Project mediawiki-core-code-coverage-docker build #4106: 09FIXED in 2 hr 7 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4106/ [17:08:07] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10Jdforrester-WMF) [17:27:59] 10Release-Engineering-Team, 10ChangeProp, 10WMF-JobQueue: Delete the mediawiki/services/cp-jobqueue repo - https://phabricator.wikimedia.org/T217683 (10Jdforrester-WMF) [17:50:13] 10Release-Engineering-Team (Kanban): Find out how WMF mobile devs do building and testing now - https://phabricator.wikimedia.org/T217688 (10LarsWirzenius) [18:05:13] 10Release-Engineering-Team (Kanban): Find out how WMF mobile devs do building and testing now - https://phabricator.wikimedia.org/T217688 (10Aklapper) (Feel free to ignore this comment.) My curiosity made me want to find public (volunteer accessible) info instead of asking co-workers. Starting on https://www.med... [18:18:37] (03PS3) 10Gehel: CI configuration for Blazegraph [integration/config] - 10https://gerrit.wikimedia.org/r/493340 (https://phabricator.wikimedia.org/T216855) (owner: 10Smalyshev) [19:11:24] (03PS1) 10Dduvall: Include a basic go.mod that declares the package name [blubber] - 10https://gerrit.wikimedia.org/r/494532 [19:13:02] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [blubber] - 10https://gerrit.wikimedia.org/r/494532 (owner: 10Dduvall) [19:16:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap in job beta-scap-eqiad has random failures: Call to mwscript eval.php returned: None - https://phabricator.wikimedia.org/T217587 (10thcipriani) p:05Triage→03High I have disabled `beta-scap-eqiad` since it's now failing const... [19:17:26] !log disable beta-scap-eqiad due to T217587 [19:17:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:17:29] T217587: scap in job beta-scap-eqiad has random failures: Call to mwscript eval.php returned: None - https://phabricator.wikimedia.org/T217587 [19:19:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Class 'Memcached' not found for php7 in beta - https://phabricator.wikimedia.org/T217587 (10thcipriani) [19:22:10] 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Legoktm) The tarballs have already been uploaded, we should just release them and plan to do the security release in a week or two. I'll help work on... [19:34:41] (03CR) 10Lars Wirzenius: [C: 03+1] "LGM" [blubber] - 10https://gerrit.wikimedia.org/r/494532 (owner: 10Dduvall) [19:34:51] (03PS2) 10Dduvall: Include a basic go.mod that declares the package name [blubber] - 10https://gerrit.wikimedia.org/r/494532 [19:37:47] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [blubber] - 10https://gerrit.wikimedia.org/r/494532 (owner: 10Dduvall) [19:39:17] 10Release-Engineering-Team, 10Growth-Team, 10MediaWiki-extensions-UserMerge, 10Notifications, and 2 others: Various tests failing with Echo/UserMerge issues - https://phabricator.wikimedia.org/T217487 (10JTannerWMF) 05Open→03Resolved a:03JTannerWMF [19:40:21] 10Release-Engineering-Team, 10Growth-Team, 10MediaWiki-extensions-UserMerge, 10Notifications, 10MW-1.33-notes (1.33.0-wmf.21; 2019-03-12): Various tests failing with Echo/UserMerge issues - https://phabricator.wikimedia.org/T217487 (10Daimona) The patch for changing Database::select was still pending, bu... [19:45:32] +/23 [19:55:38] thcipriani: for php7.2 and memcached, I felt about using a Docker container to reproduce the issue ;D [19:55:43] but err too busy ;) [20:05:57] (03CR) 10Jeena Huneidi: [C: 03+1] Include a basic go.mod that declares the package name [blubber] - 10https://gerrit.wikimedia.org/r/494532 (owner: 10Dduvall) [20:06:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Class 'Memcached' not found for php7 in beta - https://phabricator.wikimedia.org/T217587 (10Joe) `apt-get install php7.2-msgpack` solved the problem FWIW. [20:06:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Class 'Memcached' not found for php7 in beta - https://phabricator.wikimedia.org/T217587 (10Joe) 05Open→03Resolved a:03Joe [20:07:03] <_joe_> hashar: {{solved}} [20:08:23] _joe_: what was it ? :) [20:09:28] <_joe_> hashar: php-msgpack has a lower version in the php7.2-enabled version [20:09:53] how is that related to php-memcached and lack of Memcached class? :/ [20:09:54] <_joe_> which was done on purpose, but needed manual adjusting on servers that had php-fpm installed with 7.0 [20:10:01] !log reenable beta-scap-eqiad [20:10:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:10:12] <_joe_> because the memcached extension depends on the msgpack extension [20:10:22] <_joe_> if that's not available, memcached won't either [20:11:11] so I guess php-memcached debian packages lacks a Require: header [20:11:29] (not to wonder why it absolutely require msgpack :D ) [20:20:03] Project beta-update-databases-eqiad build #32206: 04FAILURE in 2.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32206/ [20:31:10] Yippee, build fixed! [20:31:10] Project beta-scap-eqiad build #240259: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240259/ [20:32:16] <_joe_> hashar: it does not miss that [20:32:40] <_joe_> hashar: I did something specifically bad in terms of package versioning, in theory [20:33:06] <_joe_> but I did it on purpose, so that we'd be notified of any security update [20:35:20] 10Beta-Cluster-Infrastructure, 10VisualEditor: Getting "Internal Error" on my sandbox on Beta cluster - https://phabricator.wikimedia.org/T217706 (10Ryasmeen) [20:38:33] (03PS1) 10Kosta Harlan: Remove requirement for properties file, run and import coverage [integration/config] - 10https://gerrit.wikimedia.org/r/494548 (https://phabricator.wikimedia.org/T215176) [20:41:29] (03CR) 10Kosta Harlan: "I wasn't sure about the change to mediawiki-extensions.yaml. When sonar-scanner runs, it needs to pick up already generated code coverage " [integration/config] - 10https://gerrit.wikimedia.org/r/494548 (https://phabricator.wikimedia.org/T215176) (owner: 10Kosta Harlan) [20:43:45] (03CR) 10Kosta Harlan: Remove requirement for properties file, run and import coverage (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/494548 (https://phabricator.wikimedia.org/T215176) (owner: 10Kosta Harlan) [20:47:59] (03CR) 10Kosta Harlan: Remove requirement for properties file, run and import coverage (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/494548 (https://phabricator.wikimedia.org/T215176) (owner: 10Kosta Harlan) [20:49:03] 10Beta-Cluster-Infrastructure, 10VisualEditor: Cannot save any edit on Beta cluster, getting error "cURL error 23: Failed writing header (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)" - https://phabricator.wikimedia.org/T217707 (10Ryasmeen) [20:58:44] _joe_: at least it is fixed! Thank you :))))) [21:17:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Score, 10VisualEditor: Getting "Internal Error" on my sandbox on Beta cluster - https://phabricator.wikimedia.org/T217706 (10Aklapper) [21:18:22] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Score, 10VisualEditor: Getting "Internal Error" on my sandbox on Beta cluster - https://phabricator.wikimedia.org/T217706 (10Reedy) [21:20:03] Project beta-update-databases-eqiad build #32207: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32207/ [21:31:03] Project beta-scap-eqiad build #240265: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240265/ [21:40:51] (03CR) 10Hashar: "The reverted commit referred to has been made with Differential and is thus not known to Gerrit. Eventually it circle back to this chang" [tools/scap] - 10https://gerrit.wikimedia.org/r/494068 (https://phabricator.wikimedia.org/T191373) (owner: 10Hashar) [21:49:09] Yippee, build fixed! [21:49:09] Project beta-scap-eqiad build #240266: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/240266/ [22:03:04] 10Phabricator: Determine policy for Phabricator multi-factor authentication reset requests - https://phabricator.wikimedia.org/T85706 (10Aklapper) 05Open→03Resolved a:03Aklapper >>! In T85706#1805699, @Aklapper wrote: > Reopning as the current policy at https://www.mediawiki.org/wiki/Phabricator/Help/Two-f... [22:06:17] (03PS1) 10C. Scott Ananian: Enable jenkins for the Zest library [integration/config] - 10https://gerrit.wikimedia.org/r/494609 [22:20:03] Project beta-update-databases-eqiad build #32208: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32208/ [22:44:39] 10Phabricator, 10Mail, 10Operations: DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org - https://phabricator.wikimedia.org/T116805 (10Niedzielski) @aklapper, {icon thumbs-up}, good to know. Thanks! [22:55:30] (03CR) 10Gergő Tisza: Enable jenkins for the Zest library (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/494609 (owner: 10C. Scott Ananian) [23:20:03] Project beta-update-databases-eqiad build #32209: 04STILL FAILING in 3.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/32209/ [23:29:26] greg-g: took the liberty to polish this graph a bit, hope you don't mind :) https://www.mediawiki.org/wiki/Version_lifecycle#Version_timeline [23:29:32] (before: https://www.mediawiki.org/w/index.php?title=Template:Timeline_MediaWiki&oldid=3051242 ) [23:29:56] Krinkle: not at all. [23:39:34] oh yes, nice [23:55:15] (03CR) 10Thcipriani: "Left some feedback inline" (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/493699 (owner: 10Effie Mouzeli)