[00:15:35] 10MediaWiki-Releasing, 10Release-Engineering-Team (Watching / External), 10Parsoid, 10Service-Architecture, 10Services (watching): Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#3873710 (10Krinkle) [00:22:06] 10MediaWiki-Releasing, 10Parsoid, 10Service-Architecture, 10Services (later): Distribution strategy option: Use Debian packages - https://phabricator.wikimedia.org/T88154#3873747 (10Krinkle) [00:38:04] 10Beta-Cluster-Infrastructure, 10MediaWiki-Language-converter: (Develop spec for) Test wiki for English language variants - https://phabricator.wikimedia.org/T143628#3873784 (10Krinkle) [00:49:18] 10Continuous-Integration-Config, 10Gerrit, 10Cleanup, 10GitHub-Mirrors, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3873797 (10Tgr) Archiving a stable extension should involve some amount of public dicsussion, not just someone making an arbit... [00:53:09] (03PS1) 10Legoktm: Add sniff to use namespaced PHPUnit\Framework\TestCase [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/401823 (https://phabricator.wikimedia.org/T184137) [00:55:58] 10Continuous-Integration-Config, 10Gerrit, 10Cleanup, 10GitHub-Mirrors, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3866385 (10demon) Just because it isn't used at WMF anymore doesn't mean it's worth archiving. I'm inclined to deny this. [00:56:13] 10Continuous-Integration-Config, 10Gerrit, 10Cleanup, 10GitHub-Mirrors, and 6 others: Archive mediawiki/extensions/Collection and others - https://phabricator.wikimedia.org/T183891#3873803 (10demon) (aka: what do the author(s) have to say?) [03:03:40] Project mwext-phpunit-coverage-publish build #22: 04FAILURE in 3 min 39 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/22/ [08:16:44] Project beta-scap-eqiad build #189280: 04FAILURE in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189280/ [08:26:38] Project beta-scap-eqiad build #189281: 04STILL FAILING in 2 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189281/ [08:36:33] Project beta-scap-eqiad build #189282: 04STILL FAILING in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189282/ [08:39:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (Someday): Switch MediaWiki coverage job from PHP 5 to PHP 7 - https://phabricator.wikimedia.org/T147778#3874489 (10Legoktm) [08:39:26] 10Continuous-Integration-Infrastructure, 10Tracking: PHP7 support in CI (tracking) - https://phabricator.wikimedia.org/T144964#3874490 (10Legoktm) [08:39:28] 10Continuous-Integration-Config, 10Release-Engineering-Team (Watching / External), 10MediaWiki-Core-Tests: MediaWiki code coverage fails on Zend PHP 7.0 due to a database error - https://phabricator.wikimedia.org/T147781#3874487 (10Legoktm) 05Open>03Resolved This was resolved at some point, running the c... [08:40:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (Someday): Switch MediaWiki coverage job from PHP 5 to PHP 7 - https://phabricator.wikimedia.org/T147778#2703156 (10Legoktm) Copying my comment from T147781#3874487: > The coverage result is a bit different though, 21.3% (44784/210421 elements) on PHP... [08:41:12] 08:36:26 08:36:26 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-tin.deployment-prep.eqiad.wmflabs::common/wikiversions*.{json,php} /srv/mediawiki on deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. [08:46:34] Project beta-scap-eqiad build #189283: 04STILL FAILING in 2 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189283/ [08:49:28] Project beta-scap-eqiad build #189284: 04STILL FAILING in 2 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189284/ [08:56:34] Project beta-scap-eqiad build #189285: 04STILL FAILING in 2 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189285/ [09:06:38] Project beta-scap-eqiad build #189286: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189286/ [09:16:40] Project beta-scap-eqiad build #189287: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189287/ [09:21:14] no_justification: Sure! I'll be here almost all day :) [09:26:37] Project beta-scap-eqiad build #189288: 04STILL FAILING in 2 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189288/ [09:36:38] Project beta-scap-eqiad build #189289: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189289/ [09:43:07] Project beta-scap-eqiad build #189290: 04STILL FAILING in 3 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189290/ [09:46:39] 10Continuous-Integration-Config, 10Operations: tox 2.5.0 on phabricator-jessie-diffs fails with ERROR: Commands not specified - https://phabricator.wikimedia.org/T184060#3874550 (10fgiunchedi) My point is more like that's a regression in tox 2.5.0 (i.e. environment without `commands` is invalid) that got rever... [09:46:51] Project beta-scap-eqiad build #189291: 04STILL FAILING in 3 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189291/ [09:56:43] Project beta-scap-eqiad build #189292: 04STILL FAILING in 3 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189292/ [10:06:51] Project beta-scap-eqiad build #189293: 04STILL FAILING in 3 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189293/ [10:13:25] 10Deployments, 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Patch-For-Review, 10User-Addshore: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3874579 (10Addshore) 05Open>03Resolved a:03Addshore [10:13:34] 10Deployments, 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Patch-For-Review, and 2 others: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3874583 (10Addshore) a:05Addshore>03Ladsgroup [10:13:43] 10Deployments, 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup: Make one group1 wiki a client of testwikidata (preferably a test wiki) - https://phabricator.wikimedia.org/T182326#3874585 (10Addshore) [10:16:52] Project beta-scap-eqiad build #189294: 04STILL FAILING in 3 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189294/ [10:26:48] Project beta-scap-eqiad build #189295: 04STILL FAILING in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189295/ [10:35:23] (03PS1) 10Legoktm: Add script to convert cobertura reports to clover [integration/jenkins] - 10https://gerrit.wikimedia.org/r/401931 (https://phabricator.wikimedia.org/T179054) [10:38:09] 10Continuous-Integration-Config, 10Wiki-Loves-Monuments-Database, 10Patch-For-Review: Generate clover.xml for labs/tools/heritage - https://phabricator.wikimedia.org/T179054#3874643 (10Legoktm) a:03Legoktm >>! In T179054#3720942, @Lokal_Profil wrote: > So generating coverage.xml I can have a patch for stra... [10:56:55] Project beta-scap-eqiad build #189296: 04STILL FAILING in 23 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189296/ [11:02:09] (03CR) 10Legoktm: [C: 032] Add script to convert cobertura reports to clover [integration/jenkins] - 10https://gerrit.wikimedia.org/r/401931 (https://phabricator.wikimedia.org/T179054) (owner: 10Legoktm) [11:03:01] (03Merged) 10jenkins-bot: Add script to convert cobertura reports to clover [integration/jenkins] - 10https://gerrit.wikimedia.org/r/401931 (https://phabricator.wikimedia.org/T179054) (owner: 10Legoktm) [11:03:13] Project beta-scap-eqiad build #189297: 04STILL FAILING in 5 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189297/ [11:07:08] Project beta-scap-eqiad build #189298: 04STILL FAILING in 3 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189298/ [11:16:51] Project beta-scap-eqiad build #189299: 04STILL FAILING in 3 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189299/ [11:26:41] Project beta-scap-eqiad build #189300: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189300/ [11:36:39] Project beta-scap-eqiad build #189301: 04STILL FAILING in 2 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189301/ [11:46:37] Project beta-scap-eqiad build #189302: 04STILL FAILING in 2 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189302/ [11:56:38] Project beta-scap-eqiad build #189303: 04STILL FAILING in 2 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189303/ [12:00:29] ooooh [12:03:57] 08:16:35 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-tin.deployment-prep.eqiad.wmflabs::common/wikiversions*.{json,php} /srv/mediawiki on deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. [12:03:57] 08:16:35 [12:04:11] looks like the scap is still succeeding though [12:06:00] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3874935 (10Addshore) [12:06:05] filed https://phabricator.wikimedia.org/T184175 [12:06:48] Project beta-scap-eqiad build #189304: 04STILL FAILING in 3 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189304/ [12:12:05] Yippee, build fixed! [12:12:06] Project mwext-phpunit-coverage-publish build #23: 09FIXED in 3 min 9 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/23/ [12:16:50] Project beta-scap-eqiad build #189305: 04STILL FAILING in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189305/ [12:26:43] Project beta-scap-eqiad build #189306: 04STILL FAILING in 3 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189306/ [12:29:43] 10Beta-Cluster-Infrastructure, 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3874981 (10mobrovac) p:05Triage>03High [12:36:38] Project beta-scap-eqiad build #189307: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189307/ [12:46:42] Project beta-scap-eqiad build #189308: 04STILL FAILING in 3 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189308/ [12:54:46] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Zuul: Add support for ecdsa keys in zuul (Also update paramiko to 2.2+) - https://phabricator.wikimedia.org/T171165#3875018 (10Paladox) Upstream have this https://review.openstack.org/#/c/528728/ change pending. [12:56:41] Project beta-scap-eqiad build #189309: 04STILL FAILING in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189309/ [13:00:51] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3874935 (10Paladox) There’s a returned [2... [13:02:20] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3875024 (10Paladox) [13:02:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday), 10Patch-For-Review, 10User-Addshore, 10Zuul: Upgrade pbr for zuul - https://phabricator.wikimedia.org/T162787#3875023 (10Paladox) [13:02:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday), 10Patch-For-Review, 10User-Addshore, 10Zuul: Upgrade pbr for zuul - https://phabricator.wikimedia.org/T162787#3174639 (10Paladox) [13:02:36] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10Paladox) [13:02:46] 10Continuous-Integration-Config, 10Technical-Debt, 10Zuul: test-requirements.txt in ci-config still points to precise deb - https://phabricator.wikimedia.org/T162191#3875027 (10Paladox) [13:02:49] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10Paladox) [13:04:17] 10Continuous-Integration-Config, 10Technical-Debt, 10Zuul: test-requirements.txt in ci-config still points to precise deb - https://phabricator.wikimedia.org/T162191#3154967 (10Paladox) I think I did it correctly. IE, do the zuul upgrade first then do this task. [13:06:39] Project beta-scap-eqiad build #189310: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189310/ [13:16:41] Project beta-scap-eqiad build #189311: 04STILL FAILING in 3 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189311/ [13:23:06] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3874935 (10ArielGlenn) I merged this; http... [13:26:40] Project beta-scap-eqiad build #189312: 04STILL FAILING in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189312/ [13:36:40] Project beta-scap-eqiad build #189313: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189313/ [13:39:47] Project beta-scap-eqiad build #189314: 04STILL FAILING in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189314/ [13:45:21] Project beta-scap-eqiad build #189315: 04STILL FAILING in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189315/ [13:49:00] Project beta-scap-eqiad build #189316: 04STILL FAILING in 2 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189316/ [13:56:43] Project beta-scap-eqiad build #189317: 04STILL FAILING in 3 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189317/ [14:06:48] Project beta-scap-eqiad build #189318: 04STILL FAILING in 3 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189318/ [14:09:58] Project beta-scap-eqiad build #189319: 04STILL FAILING in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189319/ [14:16:53] Project beta-scap-eqiad build #189320: 04STILL FAILING in 3 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189320/ [14:19:54] Project beta-scap-eqiad build #189321: 04STILL FAILING in 2 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189321/ [14:26:43] Project beta-scap-eqiad build #189322: 04STILL FAILING in 3 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189322/ [14:36:37] Project beta-scap-eqiad build #189323: 04STILL FAILING in 2 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189323/ [14:46:48] Project beta-scap-eqiad build #189324: 04STILL FAILING in 3 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189324/ [14:56:43] Project beta-scap-eqiad build #189325: 04STILL FAILING in 3 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189325/ [15:06:48] Project beta-scap-eqiad build #189326: 04STILL FAILING in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189326/ [15:08:12] Project mwext-phpunit-coverage-publish build #25: 04FAILURE in 8 min 12 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/25/ [15:11:09] Project beta-scap-eqiad build #189327: 04STILL FAILING in 3 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189327/ [15:16:46] Project beta-scap-eqiad build #189328: 04STILL FAILING in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189328/ [15:26:45] Project beta-scap-eqiad build #189329: 04STILL FAILING in 3 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189329/ [15:34:37] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3874935 (10thcipriani) >>! In T184175#3875... [15:36:51] Project beta-scap-eqiad build #189330: 04STILL FAILING in 3 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189330/ [15:41:16] "Host key verification failed." [15:41:19] Yay. [15:42:05] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3875418 (10ArielGlenn) I probably don't wa... [15:46:01] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3875427 (10ArielGlenn) role::beta::mediawi... [15:46:40] Project beta-scap-eqiad build #189331: 04STILL FAILING in 2 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189331/ [15:52:01] no_justification: can you possibly help me understand the -1 here https://integration.wikimedia.org/ci/job/operations-puppet-tests-docker/12715/console for https://gerrit.wikimedia.org/r/#/c/402068/ [15:52:10] I can't figure out if jenkins is crazy or if there is a problem I can fix [15:53:06] * no_justification has no idea [15:53:17] * no_justification 's brain is still booting [15:56:16] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Cleanup, 10Repository-Admins, 10Scoring-platform-team: Investigate and clean up deprecated ORES repos - https://phabricator.wikimedia.org/T179610#3875449 (10demon) a:05demon>03None [15:57:46] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3875458 (10ArielGlenn) BTW the reason for... [15:57:50] no_justification: I gues it was a missing ) but the verbosity there made it really hard to find [16:02:44] Weird....that a syntax error would cause cascading failures like that [16:02:45] Project beta-scap-eqiad build #189332: 04STILL FAILING in 9 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189332/ [16:12:43] Project beta-scap-eqiad build #189333: 04STILL FAILING in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189333/ [16:22:45] Project beta-scap-eqiad build #189334: 04STILL FAILING in 9 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189334/ [16:26:18] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: SCAP: Upload debian package version 3.7.4-3 - https://phabricator.wikimedia.org/T182347#3875582 (10thcipriani) [16:26:22] 10Scap, 10Operations, 10Patch-For-Review: scap 3.7.4-2 is broken - https://phabricator.wikimedia.org/T183046#3875579 (10thcipriani) 05Open>03Resolved a:03akosiaris This one was resolved with the release of scap 3.7.4-3 [16:29:42] 10Beta-Cluster-Infrastructure, 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3875601 (10thcipriani) This one seems related to https://github.com/wikimedia/scap/commit/5acb672bea5beecf677e791eb0ca332793412201 and T182865 @mmodell do you have some time to check this out? [16:30:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3875606 (10mmodell) a:03mmodell @thcipriani: I'm on it [16:32:47] Project beta-scap-eqiad build #189335: 04STILL FAILING in 9 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189335/ [16:38:40] Yippee, build fixed! [16:38:41] Project mwext-phpunit-coverage-publish build #26: 09FIXED in 3 min 24 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/26/ [16:42:52] Project beta-scap-eqiad build #189336: 04STILL FAILING in 9 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189336/ [16:52:52] Project beta-scap-eqiad build #189337: 04STILL FAILING in 9 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189337/ [17:02:48] Project beta-scap-eqiad build #189338: 04STILL FAILING in 9 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189338/ [17:04:05] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3875704 (10mmodell) [17:04:25] thcipriani: https://phabricator.wikimedia.org/D942 [17:05:15] strangely, `arc unit` is broken for me [17:05:46] does that quoting work? That seems crazy. [17:12:48] Project beta-scap-eqiad build #189339: 04STILL FAILING in 9 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189339/ [17:22:44] Project beta-scap-eqiad build #189340: 04STILL FAILING in 9 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189340/ [17:26:28] thcipriani: what seems crazy about it? [17:30:01] ¯\_(ツ)_/¯ string quoting/handling is crazy always, I guess ;) [17:30:08] sorry, was in a meeting [17:40:27] thcipriani: yeah ... I thought about implementing `git submodule foreach` in python... that might really be the best idea for handling args safely [17:43:03] what is "deployment-snapshot01.deployment-prep.eqiad.wmflabs" for? [17:43:16] it appears to be the cause of the failing 'beta-scap-eqiad' build [17:45:02] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3875835 (10mmodell) 05Open>03Resolved [17:46:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3872850 (10mmodell) indeed... our scap ci is now broken, it's not building the latest thing anymore. [17:47:11] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3875860 (10mobrovac) Thnx for the fix. Please make it a thing to test Scap in Beta when a new version lands there. [17:47:15] twentyafterfour: there's a task https://phabricator.wikimedia.org/T184175 [17:48:57] Project beta-scap-eqiad build #189341: 04STILL FAILING in 25 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189341/ [17:49:05] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3875873 (10mmodell) Perhaps we should just skip the auto-upgrade and install the testing packages in an alternate location ... but that would likely lead to less te... [17:49:07] How easy would it be to setup a second beta cluster for testing MCR on? [17:49:16] ;) [17:49:22] * thcipriani hides [17:49:35] hahhahahaa [17:49:49] without any auto updating stuff [17:49:59] 10Gerrit, 10Diffusion, 10Patch-For-Review: Commits merged in Gerrit should appear near-instantly in Phabricator - https://phabricator.wikimedia.org/T183792#3875878 (10mmodell) @chad: why does phabricator suck as a code browser? And what do you think would be better? [17:50:14] maybe we just need a mediawiki server added that we can add our own code too [17:51:02] hmmmm [17:51:18] and regularly cloned dbs or something [17:51:40] OR..... [17:52:15] create a mcrwiki on the beta cluster, and have it use a different branch that are can chuck whatever code we want into it? hmmmmm [17:52:37] addshore: not at all easy [17:52:46] wheeee :p [17:52:51] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: Scap not working in Beta - https://phabricator.wikimedia.org/T184176#3875883 (10mobrovac) >>! In T184176#3875873, @mmodell wrote: > Perhaps we should just skip the auto-upgrade and install the testing packages in an alternate locatio... [17:53:02] we've been there before, it was a bit of quagmire [17:55:00] hmm, but it 'shouldnt' be too hard to have a new wiki, with its own db, running code from its own branch? I mean, thats essentially what prod does [17:55:14] or is there some really ugly stuff I'm not thinking about ? [17:57:33] I think mostly the ability to build n betas is what we worked on a few years ago and all have some lingering twitch reaction about. [18:00:27] Project beta-scap-eqiad build #189342: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189342/ [18:03:59] thcipriani: I might throw some patches up, I dont think getting to a state that would elp us with the MCR stuff would be that much work [18:04:59] addshore: cool, having another wiki pointed to a different database doesn't seem a bridge too far. Building another beta might be :) [18:07:29] 10Gerrit, 10Diffusion, 10Patch-For-Review: Commits merged in Gerrit should appear near-instantly in Phabricator - https://phabricator.wikimedia.org/T183792#3875959 (10demon) Pagination of repos, poor search usability cross repo.... [18:09:33] addshore: that's the sort of thing that MediaWiki-Vagrant in a Cloud VPS project is made for [18:10:01] that is for example how we tested the AuthManager rewrite for months [18:10:21] Project beta-scap-eqiad build #189343: 04STILL FAILING in 9 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189343/ [18:10:28] you get a wiki or farm of wikis and the ability to easily manage cherry-picks and config [18:11:03] none of those things are easy in deployment-prep. It's a pipeline built to deploy and run the tip of master [18:15:10] eventlogging in beta cluster seems quite dead [18:15:29] disk full, schema creation looks like it stopped years ago [18:15:58] is that official abandoned or just no one noticed because no one uses it? [18:17:14] bd808: maybe I should try this vagrant in a cloud vps project stuff then! [18:17:43] bd808: is there a manual? ;) [18:18:09] addshore: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS [18:18:24] yay! [18:18:27] tgr: probably the latter [18:19:55] bd808: the authmanager code had very little interaction with how the server infrastructure is set up [18:20:14] except for centralauth, which we did break about a dozen times [18:20:15] Project beta-scap-eqiad build #189344: 04STILL FAILING in 9 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189344/ [18:20:20] tgr: sure, but it had a lot to do with pre-merge patches and config [18:20:25] the same is probably not true for MCR [18:20:55] MCR should run in mw-vagrant or it will be a mystery to develop forever ;) [18:20:56] no master-slave on vagrant, no multi-DC etc [18:21:13] although there is no multi-dc in beta either [18:21:23] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3876001 (10thcipriani) p:05Triage>03Nor... [18:22:04] I mean, sure, it should be tested on a cloud vps instance first, I just don't think that should give much confidence about the code being correct [18:22:24] none of the replag bugs show up and so on [18:24:09] Yippee, build fixed! [18:24:10] Project beta-scap-eqiad build #189345: 09FIXED in 3 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/189345/ [18:24:46] tgr: replag bugs are not likely to be caught in beta either AFAIK [18:25:09] It should be possible to add a db replica role to mw-vagrant [18:25:20] that would be T93047 [18:25:20] T93047: Create vagrant role for master-slave DB setup - https://phabricator.wikimedia.org/T93047 [18:25:24] and probably even make the replag easily tunable [18:25:44] yeah, would be nice if someone did that :) [18:25:56] tgr: so you've spent 2 years not writing it? ;) [18:26:07] I guess you are right, vagrant is better for that in theory than beta [18:27:49] In a more perfect world it would be easier to have N deployment-prep like environments. But in that world Cloud VPS would also have a lot more hardware and RelEng would have 2 roots [19:24:43] bd808: let's do it. [19:25:04] greg-g: :) works for me [19:32:58] thcipriani: o/, so I tried to deploy mobileapps to beta cluster earlier and it failed miserably. Talking about `git submodule foreach git tag -fa `[...] and `Entering 'src'` and `fatal: bad object type` [19:35:52] bearND: hrm, I think that's a new one. [19:36:52] thcipriani: You probably see it in the log file and I don't need to post the full stack trace, right? [19:37:10] twentyafterfour: ^ I think your quoting patch may have missed one [19:37:20] bearND: yeah, I was just looking at scap deploy-log [19:37:42] thcipriani: probably just not deployed [19:37:44] our deployment stalking feature :) [19:37:57] scap auto deploy to beta is broken [19:38:43] newest package has today's date FWIW: 3.8.0-1~20180104174716.269 [19:40:02] hmm it's not quoting related, this is a different thing [19:45:06] thcipriani: bearND: https://phabricator.wikimedia.org/D943 [19:48:39] twentyafterfour: think I'm seeing a different error now, commented. [19:51:04] 10Gerrit, 10Diffusion, 10Patch-For-Review: Commits merged in Gerrit should appear near-instantly in Phabricator - https://phabricator.wikimedia.org/T183792#3876364 (10mmodell) I agree about search, however, [[ https://codesearch.wmflabs.org | codesearch ]] helps with that situation [19:54:07] thcipriani: that's...odd [19:54:44] yeah, no idea where -c is coming from... [19:56:15] clean_tags I guess [19:57:26] oh I see [19:57:27] updated [19:57:43] the -c is just wrong [19:57:45] afaikt [19:58:04] I need to get my scap testing vm working agin [19:58:08] but right now, train [19:58:45] heh > error: tag 'scap/sync/2017-12-11/0001' not found [20:03:13] updated again [20:06:41] same error. We didn't used to tag the submodules. I think it's finding old tags from the main repo and trying to delete them in the submodules. [20:06:58] since it's not doing the submodule foreach thing [20:07:13] right, I added ||true because that should make it succeed ;) [20:07:55] I guess the train is blocked for now so I'll set up scap test environment again so that you don't have to play CI for me [20:08:19] I mean, not that you aren't doing an excellent job and everything...:D [20:08:50] 10Release-Engineering-Team (Kanban), 10Scap: Scap canary has a shifting baseline - https://phabricator.wikimedia.org/T183999#3876431 (10Krinkle) @mmodell A dynamic query for 24h ago seems somewhat fragile given that 1) traffic volume has strong seasonality (varying upto 4x on a regular basis), and 2) 24h could... [20:09:37] twentyafterfour: Regarding T183999, the task description mentions "since the last deploy". Does that mean we compare to the rate from the canaries found during the last deploy (e.g. a number fixed stored somewhere in a local log?) Or do we query live Logstash for the time window right before the current deploy attempt? [20:09:38] T183999: Scap canary has a shifting baseline - https://phabricator.wikimedia.org/T183999 [20:09:44] I assumed the latter [20:10:46] fwiw, in mobileapps we add tags to the source repo manually after we deploy to prod, so we know what new stuff is to be deployed. Would be great to have this automated. [20:11:38] Krinkle: yeah, your assumption is correct [20:28:46] no_justification aha https://gerrit-review.googlesource.com/#/c/gerrit/+/150831/ [20:37:02] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3876481 (10ArielGlenn) That's fine. I've... [20:56:32] hm, I keep missing hashar [21:00:14] legoktm: work on your aim [21:00:31] twentyafterfour: Okay. Thanks :) [21:00:48] heh [21:01:10] (03CR) 10Umherirrender: [C: 031] "I have not checked, if the extension has dependency to other extensions. But having non-voting tests at all would be nice." [integration/config] - 10https://gerrit.wikimedia.org/r/394578 (owner: 10Robert Vogel) [21:02:18] twentyafterfour: Storing the rate itself after a good prod-wide sync could also be interesting. But has the downside of potentially comparing different times of day which can be problematic unless we somehow divide the error volume by # of requests or some such. Also, if we store the rate itself, we could cause a deploy to "fail" due to an increased rate that doesn't caused by a deploy but built up since the last deploy, which could [21:02:19] be annoying. [21:20:18] 10Continuous-Integration-Infrastructure, 10Composer, 10Patch-For-Review: Upgrade integration/composer to 1.4.3 stable - https://phabricator.wikimedia.org/T125343#3876563 (10Legoktm) We're pretty behind, and probably should jump to 1.6.x now... @hashar reverted this last time noting that we also had to annou... [21:30:32] Krinkle: it already normalizes the error rate, I think [21:30:47] it's looking at the ratio of errors to successes [21:31:08] so gross request volume shouldn't confuse it [21:38:16] grr... phan job is failing on my TitleBlacklist patch: https://gerrit.wikimedia.org/r/#/c/402108/ [21:39:50] twentyafterfour: I'm looking into it now, but don't let it block the train, the warning is the issue. Also, can you use the normal Bug: syntax so all the tools pick up it properly? nothing recognizes 'refs" [21:41:05] twentyafterfour: you can cherry-pick https://gerrit.wikimedia.org/r/#/c/401655/ [21:41:09] legoktm: ok. So can I v+2 it then to override the test fail? [21:41:17] no [21:41:21] use the other patch [21:41:30] ah ok [21:41:31] thanks [21:41:45] also, not submitting patches against master is really problematic [21:41:50] because no one follows-up on them :( [21:41:57] I had to fix a fatal last night because of that [21:42:23] well this was just logspam so I wasn't concerned about it causing problems, and I wanted to get at least code review before submitting it to master, that's all [21:42:43] in general I totally agree it should go to master [21:44:43] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3876589 (10ArielGlenn) With some assist fr... [21:52:33] legoktm: thanks for your help! [21:52:55] np, and sorry about for letting that slip through in CR :( [22:01:24] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-scap-eqiad FAILED - deployment-snapshot01.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. - https://phabricator.wikimedia.org/T184175#3876623 (10thcipriani) 05Open>03Resolve... [22:07:26] Can someone please finally update my deployment-prep root key? [22:14:07] legoktm? [22:15:06] twentyafterfour? [22:15:26] Krenair: root key? [22:15:30] yes [22:15:52] link? [22:15:54] https://phabricator.wikimedia.org/P6209 [22:19:12] * twentyafterfour is trying to figure out where it goes [22:19:16] is there a puppet module? [22:19:49] https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep [22:21:41] ah ok. done [22:22:07] thanks twentyafterfour [22:23:58] works [22:30:49] anyone know what's up with deployment-prep? even my old root key doesn't work there, much less my new one, or cumin for that matter [22:31:42] deployment-phab* [22:32:04] i think puppet is broken [22:32:06] on that host [22:32:29] hashar was mentioned that when he upgraded puppet to 4 on deployment-prep. [22:34:21] must've been broken from the beginning or a very long time ago? [22:42:35] jeez this project is a mess [22:45:21] which? [22:46:04] deployment-prep [22:46:13] * Hauskatze trying to find someone with terbium access to run a script [22:46:14] it looks like no one has been taking care of puppet [22:46:40] Krenair: I've been taking care of spam there, but I don't touch things I don't know :) [22:46:45] Hauskatze: finding them is not normally the hard part [22:47:03] some namespace dupes as well, etc; but nothing beyond that [22:47:29] yeah no one expects you to take care of puppet in deployment-prep Hauskatze [22:48:14] coolio, 'cause I might very well take those sites down accidentally :P [22:49:19] * Hauskatze eyes thcipriani - wonders how the category reindexing for eswiki is going... [22:50:37] Hauskatze: category reindexing for eswiki is all done [22:50:40] finished by the time I checked it this morning 7ish hours ago [22:51:23] thcipriani: thanks very much :) I wonder if you could have a minute for T184129#3875002 ? [22:51:23] T184129: Large number of broken 'pagelinks' on eswiki - https://phabricator.wikimedia.org/T184129 [22:51:49] my temporary adminship on eswiki is expiring and if there are more things to delete I'd like to take care of them now if at all possible [22:52:04] sure, lemme try that script again and get you the output [22:52:14] thanks sir [22:55:43] Hauskatze: update: https://phabricator.wikimedia.org/P6522#36740 [22:56:31] thcipriani: that was with --fix ? [22:56:38] yep [22:56:52] so nothing else needs taking care of? [22:58:00] 0 pages to fix seems good, script seems happy. [22:58:16] I breath alleviated [22:58:22] thanks so much [22:58:27] :) [23:00:21] 10Gerrit, 10Diffusion, 10Patch-For-Review: Commits merged in Gerrit should appear near-instantly in Phabricator - https://phabricator.wikimedia.org/T183792#3876699 (10demon) Well, that's the point....as nice as the tool is, it's annoying that we need an external tool.... Plus, my main point is regarding git... [23:01:03] 10Gerrit, 10Diffusion, 10Patch-For-Review: Commits merged in Gerrit should appear near-instantly in Phabricator - https://phabricator.wikimedia.org/T183792#3876700 (10demon) I don't really care what repo viewer people use //generally/./, but for the links within Gerrit, Phab is a pretty annoying context switch. [23:06:36] I see at least 13 deployment-prep hosts with puppet obviously broken in some way [23:07:34] sorry, 18 [23:13:43] Project beta-code-update-eqiad build #188149: 04FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/188149/ [23:14:22] Krenair: doesn't look good [23:14:29] what's exactly wrong with them? [23:14:34] various things [23:15:20] OOM, disk full, role KIA, random puppet disables from november, systemd vs. upstart, package missing, missing hieradata [23:15:33] all stuff I can't believe no one has noticed [23:15:40] systemd vs. upstart is due to the host being trusty [23:16:26] no icinga notices for beta cluster? [23:16:55] paladox, correct [23:17:11] Hauskatze, used to have shinken [23:17:35] who knows where that got to [23:18:37] sorry for the cheap humour, but maybe shinken... sinked [23:18:39] or sunk [23:18:46] damn English verbs [23:18:50] yep [23:21:29] well [23:21:34] that did the trick [23:23:30] Project beta-code-update-eqiad build #188150: 04STILL FAILING in 30 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/188150/ [23:25:05] ooooooo [23:33:35] Project beta-code-update-eqiad build #188151: 04STILL FAILING in 34 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/188151/ [23:35:31] (03PS1) 10Legoktm: doc: Add link for Wikidata Query Service [integration/docroot] - 10https://gerrit.wikimedia.org/r/402141 [23:35:48] (03CR) 10Legoktm: [C: 032] doc: Add link for Wikidata Query Service [integration/docroot] - 10https://gerrit.wikimedia.org/r/402141 (owner: 10Legoktm) [23:36:25] (03Merged) 10jenkins-bot: doc: Add link for Wikidata Query Service [integration/docroot] - 10https://gerrit.wikimedia.org/r/402141 (owner: 10Legoktm) [23:36:31] (03CR) 10jenkins-bot: doc: Add link for Wikidata Query Service [integration/docroot] - 10https://gerrit.wikimedia.org/r/402141 (owner: 10Legoktm) [23:37:50] good night Krenair; hope that those puppet issues can be easily resolved [23:37:57] * Hauskatze -> bed [23:38:03] g'night [23:43:36] Project beta-code-update-eqiad build #188152: 04STILL FAILING in 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/188152/ [23:45:54] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Method for running e2e/smoke tests on deployments - https://phabricator.wikimedia.org/T184219#3876762 (10dduvall) [23:46:16] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:46:38] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Method for running e2e/smoke tests on deployments - https://phabricator.wikimedia.org/T184219#3876773 (10dduvall) p:05Triage>03Normal [23:48:28] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3876789 (10dduvall) p:05Triage>03Normal [23:48:53] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3845669 (10dduvall) [23:48:56] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Method for running e2e/smoke tests on deployments - https://phabricator.wikimedia.org/T184219#3876791 (10dduvall) [23:50:56] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Build service-checker image for use with helm test - https://phabricator.wikimedia.org/T184220#3876797 (10dduvall) p:05Triage>03Normal [23:53:34] Project beta-code-update-eqiad build #188153: 04STILL FAILING in 34 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/188153/ [23:55:01] 10Continuous-Integration-Config, 10RelEng-Archive-FY201718-Q2, 10Release-Engineering-Team (Kanban), 10Composer, 10Patch-For-Review: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3876840 (10Tgr) I just got this error on a local vagran... [23:55:11] 10Beta-Cluster-Infrastructure: various .beta.wmflabs.org domains use an invalid ssl certificate - https://phabricator.wikimedia.org/T182927#3876841 (10Krenair) >>! In T182927#3860911, @Krenair wrote: > https://letsencrypt.org/2017/12/07/looking-forward-to-2018.html >>First, we’re planning to introduce an ACME v2... [23:58:49] legoktm: Kudos on cover/extensions. Very nice :) [23:58:58] thanks :D [23:59:40] legoktm: Wanna add to doc.wm.o nav menu? [23:59:52] yeah, I wasn't sure where to