[00:06:04] Krinkle: Yes. Working on having the selenium tests at least run on buster. [01:25:37] (03PS1) 10Krinkle: doc: Add navtiming entry [integration/docroot] - 10https://gerrit.wikimedia.org/r/594336 [01:25:50] (03PS2) 10Krinkle: doc: Add navtiming entry [integration/docroot] - 10https://gerrit.wikimedia.org/r/594336 (https://phabricator.wikimedia.org/T251213) [01:29:12] (03PS3) 10Krinkle: doc: Add navtiming entry [integration/docroot] - 10https://gerrit.wikimedia.org/r/594336 (https://phabricator.wikimedia.org/T251213) [01:35:01] (03PS4) 10Krinkle: doc: Add navtiming entry [integration/docroot] - 10https://gerrit.wikimedia.org/r/594336 (https://phabricator.wikimedia.org/T251213) [01:35:04] (03CR) 10Krinkle: [C: 03+2] doc: Add navtiming entry [integration/docroot] - 10https://gerrit.wikimedia.org/r/594336 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle) [01:35:44] (03Merged) 10jenkins-bot: doc: Add navtiming entry [integration/docroot] - 10https://gerrit.wikimedia.org/r/594336 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle) [04:36:20] (03PS1) 10Krinkle: doc: Add entry for performance.wikimedia.org and some other sites [integration/docroot] - 10https://gerrit.wikimedia.org/r/594354 (https://phabricator.wikimedia.org/T251213) [04:36:51] (03CR) 10Krinkle: [C: 03+2] doc: Add entry for performance.wikimedia.org and some other sites [integration/docroot] - 10https://gerrit.wikimedia.org/r/594354 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle) [04:37:19] (03Merged) 10jenkins-bot: doc: Add entry for performance.wikimedia.org and some other sites [integration/docroot] - 10https://gerrit.wikimedia.org/r/594354 (https://phabricator.wikimedia.org/T251213) (owner: 10Krinkle) [05:32:51] 10MediaWiki-Codesniffer, 10User-DannyS712: Add `goto` to AlternativeSyntaxSniff - https://phabricator.wikimedia.org/T251783 (10thiemowmde) For reference: We disallow `goto` since 2015. There is Generic.PHP.DiscourageGoto which we use since https://gerrit.wikimedia.org/r/398671. Before, we even had a custom sni... [06:06:45] Yippee, build fixed! [06:06:46] Project mwcore-phpunit-coverage-master build #645: 09FIXED in 3 hr 6 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/645/ [06:27:54] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:16] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:08] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52086 bytes in 0.978 second response time [09:03:48] 10Gerrit, 10Operations, 10vm-requests, 10Patch-For-Review: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) Icinga monitoring for gerrit1002 has been removed. Thanks Cole. [10:21:56] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing: Confirm 'vendor' job for MW installs uses prod deps correctly - https://phabricator.wikimedia.org/T251819 (10hashar) ;-] [10:27:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10hashar) [11:38:54] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:13] 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions, 10Documentation: Generate Doxygen documentation for Flow PHP classes to doc.wikimedia.org - https://phabricator.wikimedia.org/T93107 (10hashar) [12:11:15] 10Continuous-Integration-Config, 10Documentation, 10Patch-For-Review: Jenkins: Set up generated php documentation for MediaWiki extensions - https://phabricator.wikimedia.org/T27978 (10hashar) [12:11:17] 10Continuous-Integration-Config, 10Growth-Team, 10Notifications, 10Documentation: Generate doxygen for Echo on doc.wm.o - https://phabricator.wikimedia.org/T117025 (10hashar) [12:11:42] 10Continuous-Integration-Config, 10Documentation, 10Patch-For-Review: Jenkins: Set up generated php documentation for MediaWiki extensions - https://phabricator.wikimedia.org/T27978 (10hashar) Repositories having a `Doxyfile`: extensions/ChessBrowser/Doxyfile extensions/CirrusSearch/Doxyfile extensions/Coll... [12:38:43] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93090 bytes in 1.041 second response time [12:38:46] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92780 bytes in 1.011 second response time [12:39:39] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [12:40:03] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52087 bytes in 1.211 second response time [13:39:53] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:07] (03PS1) 10Hashar: jjb: import operations-puppet-catalog-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/594479 (https://phabricator.wikimedia.org/T97513) [13:41:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:26] (03CR) 10Hashar: [C: 03+2] jjb: import operations-puppet-catalog-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/594479 (https://phabricator.wikimedia.org/T97513) (owner: 10Hashar) [13:41:32] 10Phabricator, 10I18n: Add Turkish (tr) language to Phabricator - https://phabricator.wikimedia.org/T244634 (10Baris6161TURK) 05Open→03Resolved [13:41:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Developer Productivity, 10Technical-Debt: Automatically delete old jobs not (or no longer) managed by JJB - https://phabricator.wikimedia.org/T91410 (10hashar) [13:41:55] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Have jenkins jobs logrotate their build history - https://phabricator.wikimedia.org/T91396 (10hashar) [13:41:57] 10Continuous-Integration-Config, 10puppet-compiler, 10Patch-For-Review: Migrate Jenkins job "operations-puppet-catalog-compiler" to Jenkins Job Builder - https://phabricator.wikimedia.org/T97513 (10hashar) 05Open→03Resolved a:03hashar The job is now defined in jjb. [13:42:28] (03Merged) 10jenkins-bot: jjb: import operations-puppet-catalog-compiler [integration/config] - 10https://gerrit.wikimedia.org/r/594479 (https://phabricator.wikimedia.org/T97513) (owner: 10Hashar) [13:49:46] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92779 bytes in 0.976 second response time [13:49:46] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93089 bytes in 1.029 second response time [13:51:06] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52087 bytes in 1.193 second response time [14:11:18] 10Continuous-Integration-Config, 10Composer: integration/composer need a integration-composer-check-hhvm job - https://phabricator.wikimedia.org/T158845 (10hashar) 05Open→03Declined We are phasing out `integration/composer` in favor of using upstream phar archive ( T249949 ) [14:12:07] 10Continuous-Integration-Infrastructure: Stop using integration/composer and then archive the repo - https://phabricator.wikimedia.org/T249949 (10hashar) For the CI containers we went to use upstream `composer.phar`. For deployment of composer on WMCS instances via puppet, it seems to me a Debian package will b... [14:12:27] 10Continuous-Integration-Infrastructure: Stop using integration/composer and then archive the repo - https://phabricator.wikimedia.org/T249949 (10hashar) [14:12:29] 10LibUp: Use packaged composer - https://phabricator.wikimedia.org/T249525 (10hashar) [14:16:07] 10LibUp: Use packaged composer - https://phabricator.wikimedia.org/T249525 (10hashar) 05Stalled→03Open Debian provides: | Stretch | 1.2.2 | Buster | 1.8.4 | testing | 1.10.5 | unstable | 1.10.5 We can fork import their 1.10.5 packaging and rebuild it for Stretch / Buster then upload that to apt.wikimedia.org. [14:16:09] 10Continuous-Integration-Infrastructure: Stop using integration/composer and then archive the repo - https://phabricator.wikimedia.org/T249949 (10hashar) [14:18:04] 10LibUp: Make a bot to bump our trivial-but-with-security-warnings Ruby CI dependency chain - https://phabricator.wikimedia.org/T181576 (10hashar) [14:21:49] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Core-Testing, 10Vue.js: Use Jest in MediaWiki CI to test Vue.js components and plugins - https://phabricator.wikimedia.org/T251790 (10hashar) Isn't it a duplicate of @Niedzielski task {T249301} ? [14:33:01] (03PS1) 10Mholloway: Add service pipeline testing for mediawiki/services/push-notifications [integration/config] - 10https://gerrit.wikimedia.org/r/594495 (https://phabricator.wikimedia.org/T251901) [14:33:11] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10MediaWiki-Core-Testing, 10Vue.js: Use Jest in MediaWiki CI to test Vue.js components and plugins - https://phabricator.wikimedia.org/T251790 (10Jdforrester-WMF) >>! In T251790#6108862, @hashar wrote: > Isn't it a... [14:33:41] (03CR) 10Jforrester: [C: 03+2] "Ha, beat me to it." [integration/config] - 10https://gerrit.wikimedia.org/r/594495 (https://phabricator.wikimedia.org/T251901) (owner: 10Mholloway) [14:34:40] (03Merged) 10jenkins-bot: Add service pipeline testing for mediawiki/services/push-notifications [integration/config] - 10https://gerrit.wikimedia.org/r/594495 (https://phabricator.wikimedia.org/T251901) (owner: 10Mholloway) [14:36:21] !log Zuul: Install service pipeline for mediawiki/services/push-notifications T251901 [14:36:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:36:24] T251901: Set up CI for the push-notifications service - https://phabricator.wikimedia.org/T251901 [14:38:08] 10LibUp: Use packaged composer - https://phabricator.wikimedia.org/T249525 (10Reedy) >>! In T249525#6108843, @hashar wrote: > Debian provides: > > | Stretch | 1.2.2 > | Buster | 1.8.4 > | testing | 1.10.5 > | unstable | 1.10.5 > > We can fork import their 1.10.5 packaging and rebuild it for Stretch / Buster th... [14:40:40] 10Continuous-Integration-Config: Make test vote Verified+1 instead of +2 to avoid unintentional submit - https://phabricator.wikimedia.org/T226123 (10hashar) [14:58:45] (03PS5) 10Thcipriani: train-deploy-notes: change pipeline trigger [integration/config] - 10https://gerrit.wikimedia.org/r/589163 [15:01:03] (03CR) 10Jforrester: [C: 03+2] "Let's try this." [integration/config] - 10https://gerrit.wikimedia.org/r/589163 (owner: 10Thcipriani) [15:02:02] (03Merged) 10jenkins-bot: train-deploy-notes: change pipeline trigger [integration/config] - 10https://gerrit.wikimedia.org/r/589163 (owner: 10Thcipriani) [15:02:27] !log Zuul: Change pipeline trigger for train-deploy-notes job [15:02:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:02:46] James_F: <3 thanks [15:03:22] thcipriani: Let's hope it works. :-) [15:03:41] heh, "this time for sure" [15:03:46] * James_F should probably stalk the zuul log when release-jenkins is triggered. [15:09:26] 10Diffusion, 10Tool-spacemedia, 10Wikimedia-GitHub, 10cloud-services-team (Kanban): Diffusion mirroring to GitHub doesn't work for tool-spacemedia - https://phabricator.wikimedia.org/T251699 (10bd808) I updated the docs at https://wikitech.wikimedia.org/wiki/Toolsadmin.wikimedia.org/FAQ#Mirror_a_Diffusion_... [15:21:26] 10LibUp: Use packaged composer - https://phabricator.wikimedia.org/T249525 (10hashar) We do since integration/composer has to be phased out but is still used by other applications outside of CI :] [15:21:46] 10Continuous-Integration-Infrastructure: Stop using integration/composer and then archive the repo - https://phabricator.wikimedia.org/T249949 (10hashar) p:05Triage→03Medium [15:25:18] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-CodeReview, 10Wikimedia-Site-requests, and 2 others: Undeploy CodeReview - https://phabricator.wikimedia.org/T116948 (10Jdforrester-WMF) [15:30:16] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-CodeReview, 10Wikimedia-Site-requests, and 2 others: Undeploy CodeReview - https://phabricator.wikimedia.org/T116948 (10jcrespo) T243055 is done, but I am guessing dropping the table data is blocked on T205361? [15:31:54] (03PS1) 10Hashar: Fail when repository lacks gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/594507 [15:32:54] (03CR) 10jerkins-bot: [V: 04-1] Fail when repository lacks gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/594507 (owner: 10Hashar) [15:32:56] (03PS3) 10Krinkle: [WIP] Draft Jenkins job for running tests inside a sub-VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [15:34:48] hashar: yikes, where outside CI? [15:34:54] (composer.git) [15:35:00] Krinkle: Cloud [15:35:14] I can't remember where exactly... But some places are using that repo [15:37:09] (03PS2) 10Hashar: Fail when repository lacks gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/594507 [15:37:11] (03PS1) 10Hashar: tests: sort output of some assertion [integration/config] - 10https://gerrit.wikimedia.org/r/594508 [15:38:09] (03CR) 10jerkins-bot: [V: 04-1] Fail when repository lacks gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/594507 (owner: 10Hashar) [15:43:37] (03PS1) 10Hashar: jjb: puppet compiler requires a COMPILER_MODE [integration/config] - 10https://gerrit.wikimedia.org/r/594511 (https://phabricator.wikimedia.org/T97513) [15:43:58] (03CR) 10Hashar: [C: 03+2] "Deployed, that can be seen at https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/build" [integration/config] - 10https://gerrit.wikimedia.org/r/594511 (https://phabricator.wikimedia.org/T97513) (owner: 10Hashar) [15:44:49] (03Merged) 10jenkins-bot: jjb: puppet compiler requires a COMPILER_MODE [integration/config] - 10https://gerrit.wikimedia.org/r/594511 (https://phabricator.wikimedia.org/T97513) (owner: 10Hashar) [15:51:45] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-CodeReview, 10Wikimedia-Site-requests, and 2 others: Undeploy CodeReview - https://phabricator.wikimedia.org/T116948 (10Jdforrester-WMF) >>! In T116948#6109114, @jcrespo wrote: > T243055 is done, but I am gue... [16:07:18] 10MediaWiki-Codesniffer, 10Composer: MediaWiki-Codesniffer test files not complying with PSR-4 - https://phabricator.wikimedia.org/T251913 (10Tgr) [16:14:29] (03PS1) 10Jforrester: Release 0.0.43 [integration/quibble] - 10https://gerrit.wikimedia.org/r/594517 [16:14:31] (03PS1) 10Jforrester: changelog: Begin new 0.0.44 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/594518 [16:16:20] (03PS2) 10Jforrester: Release 0.0.43 [integration/quibble] - 10https://gerrit.wikimedia.org/r/594517 [16:16:22] (03PS2) 10Jforrester: changelog: Begin new 0.0.44 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/594518 [16:16:31] (03CR) 10Jforrester: [C: 03+2] Release 0.0.43 [integration/quibble] - 10https://gerrit.wikimedia.org/r/594517 (owner: 10Jforrester) [16:16:38] (03CR) 10Jforrester: [C: 03+2] changelog: Begin new 0.0.44 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/594518 (owner: 10Jforrester) [16:17:32] (03Merged) 10jenkins-bot: Release 0.0.43 [integration/quibble] - 10https://gerrit.wikimedia.org/r/594517 (owner: 10Jforrester) [16:17:34] (03Merged) 10jenkins-bot: changelog: Begin new 0.0.44 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/594518 (owner: 10Jforrester) [16:38:56] 10Release-Engineering-Team, 10Gerrit-Privilege-Requests, 10Pywikibot, 10Pywikibot-tests: Grant Pywikibot-gerritbot an access to Gerrit stream (stream-events group) - https://phabricator.wikimedia.org/T248262 (10Urbanecm) >>! In T248262#6083047, @Legoktm wrote: > Approved from my perspective. @Dvorapa can y... [16:42:09] (03PS4) 10Jforrester: jjb: [parsoidsvc-parsertests-docker] Update path to parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/588026 (https://phabricator.wikimedia.org/T248726) (owner: 10C. Scott Ananian) [16:42:11] (03PS1) 10Jforrester: dockerfiles: [quibble-stretch] Install quibble 0.0.43 and cascade. [integration/config] - 10https://gerrit.wikimedia.org/r/594521 [16:42:13] (03PS1) 10Jforrester: jjb: Upgrade the world to quibble 0.0.43 [integration/config] - 10https://gerrit.wikimedia.org/r/594522 [16:42:32] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [quibble-stretch] Install quibble 0.0.43 and cascade. [integration/config] - 10https://gerrit.wikimedia.org/r/594521 (owner: 10Jforrester) [16:43:24] (03Merged) 10jenkins-bot: dockerfiles: [quibble-stretch] Install quibble 0.0.43 and cascade. [integration/config] - 10https://gerrit.wikimedia.org/r/594521 (owner: 10Jforrester) [16:44:56] 10Continuous-Integration-Config: Make test vote Verified+1 instead of +2 to avoid unintentional submit - https://phabricator.wikimedia.org/T226123 (10hashar) [16:54:15] 10phan-taint-check-plugin, 10Patch-For-Review: Understand what's going on with ChangeTags::modifyDisplayQuery - https://phabricator.wikimedia.org/T250689 (10Daimona) a:03Daimona [17:07:28] hashar: Is it OK for me to cherry-pick upstream zuul patches into integration/zuul? [17:07:51] James_F: which ones ?;D [17:08:10] hashar: The ones getting rid of the "WARNING zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit." spam. :-) [17:08:16] it is being repurposed to use scap to deploy [17:08:24] Trivial little patches, apply cleanly. [17:08:37] Right. [17:08:44] But from our custom repo still? [17:09:12] contint2001 has it installed using integration/zuul/deploy which has integration/zuul as a submodule [17:09:21] that is deployed and used to run the zuul-merger [17:09:32] * James_F nods. [17:09:55] I am hoping to switch the zuul scheduler from contint1001 to contint2001 next week. Gotta find a date/time and announce it [17:10:01] then I guess we can ccherry pick stuff [17:10:15] OK, I'll propose them and leave them unmerged. [17:10:24] that WARNING is merely log spam anyway, it does not do any harm [17:10:37] I know, but it's a log I look at carefully. [17:10:41] though potentially we could mute it by changing the log level for zuul.GerritEventConnector [17:10:42] (03PS1) 10Jforrester: Squelch ref-replication gerrit warnings [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/594526 [17:10:44] (03PS1) 10Jforrester: Add entry for ref-replication-scheduled event [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/594527 [17:11:07] lets hold that until zuul is 100% running out of the scap one [17:11:19] so then I can phase out the patch-queue.* / debian packaging stuff [17:12:49] Hi everyone. Is this where I can ask for a particular change to be added to a SWAT window? [17:12:54] James_F: that class has only two logs, that warning and an exception logging which is at error level [17:13:17] James_F: so we can just mute zuul.GerritEventConnector :) [17:13:44] huji: I guess you can probably just add it to https://wikitech.wikimedia.org/wiki/Deployments [17:13:58] huji: it helps to have the patch reviewed ahead of time [17:14:24] hashar: And then move to zuul3? ;-) [17:14:31] James_F: ;] [17:14:37] * James_F coughs. [17:15:22] hashar: I asked here because that page says: To schedule a deploy window, or if you see a potential conflict with your upcoming deployment, please contact the Release Engineering Team on IRC at #wikimedia-releng connect; pinging Greg (greg-g) or Tyler (thcipriani) directly helps. [17:16:36] window vs scheduling in SWAT. SWAT is self-service. Another whole window you should ask us here. [17:18:34] Ahhhh... [17:24:36] Also, super low priority, but the channel log link up there is incorrect. The correct URL is http://bots.wmflabs.org/browser/index.php?display=%23wikimedia-releng [17:24:56] (03CR) 10Hashar: "+1 but lets hold until the production service has switched to the scap deployed version ;)" [integration/zuul] (patch-queue/debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/594526 (owner: 10Jforrester) [17:25:34] * Huji says byeeee [17:51:40] 10Continuous-Integration-Config, 10Operations, 10serviceops: docker-reporter-releng-images failed on deneb - https://phabricator.wikimedia.org/T251918 (10hashar) [17:52:39] (03CR) 10Brennen Bearnes: [C: 03+2] values.example.yaml: Rename restbase to restrouter [releng/local-charts] - 10https://gerrit.wikimedia.org/r/593304 (owner: 10Jeena Huneidi) [17:54:04] (03CR) 10Jeena Huneidi: [V: 03+2] values.example.yaml: Rename restbase to restrouter [releng/local-charts] - 10https://gerrit.wikimedia.org/r/593304 (owner: 10Jeena Huneidi) [18:18:24] (03PS4) 10Krinkle: [WIP] Draft Jenkins job for running tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [18:18:30] (03PS5) 10Krinkle: [WIP] Draft Jenkins job for running tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [18:19:15] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Draft Jenkins job for running tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) (owner: 10Krinkle) [18:27:10] (03PS2) 10Thiemo Kreuz (WMDE): Don't run parallel-lint on test files [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/592947 [18:27:15] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Research available third-party Argo and Kubernetes cloud providers - https://phabricator.wikimedia.org/T244384 (10dduvall) 05Open→03Stalled a:05dduvall→03None [18:27:17] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Security-Team: Refine Seakeeper proposal for Security/SRE review - https://phabricator.wikimedia.org/T243436 (10dduvall) [18:27:29] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Security-Team: Refine Seakeeper proposal for Security/SRE review - https://phabricator.wikimedia.org/T243436 (10dduvall) 05Open→03Stalled a:05dduvall→03None [18:27:41] (03PS1) 10Jforrester: [WIP] make-release: Set the right branch point for sub-repos [tools/release] - 10https://gerrit.wikimedia.org/r/594545 [18:35:05] 10Release-Engineering-Team (CI & Testing services), 10Release Pipeline: Experiment with different PipelineLib-/helm-based approaches to system testing - https://phabricator.wikimedia.org/T244313 (10dduvall) a:05dduvall→03None [18:38:40] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10wikimedia.biterg.io: Investigate wikimedia.biterg.io as possible replacement for people.wikimedia.org/~dduvall/jenkins - https://phabricator.wikimedia.org/T248351 (10dduvall) a:05dduvall→03None [19:05:17] !log Docker: Publishing quibble 0.0.43 images. [19:05:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:05:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:48] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92779 bytes in 1.003 second response time [19:18:08] 10Scap, 10User-brennen: Investigate scap-cdb-rebuild idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10mmodell) >>! In T223287#5500980, @hashar wrote: > Most probably related to imported `scap/sh.py` which probably should be removed T222372 what makes you think it's `sh.py`... [19:27:17] 10Scap, 10User-brennen: Investigate scap-cdb-rebuild idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10mmodell) it seems like this isn't just happening in cdb-rebuild but all of the cases where scap does a cluster_ssh loop. [19:27:39] 10Scap, 10User-brennen: Investigate scap cluster_ssh idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10mmodell) [19:31:09] 10Scap, 10User-brennen: Investigate scap cluster_ssh idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10mmodell) I did some more reading of the scap code and it doesn't look like `sh.py` is in the code paths where this is happening, so I really don't think `sh.py` is the prob... [19:32:02] 10Scap, 10User-brennen: Investigate scap cluster_ssh idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10thcipriani) >>! In T223287#6110404, @mmodell wrote: > it seems like this isn't just happening in cdb-rebuild but all of the cases where scap does a cluster_ssh loop. One i... [19:38:00] 10Scap, 10User-brennen: Investigate scap cluster_ssh idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10mmodell) @thcipriani yeah, I strongly suspect ssh over scap proper. one thing that //might// be related is this: We pass these ssh_config options on the command line, app... [19:43:00] 10Scap, 10User-brennen: Investigate scap cluster_ssh idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10thcipriani) >>! In T223287#6110438, @mmodell wrote: > Have we always been using these debian specific options? `identitiesOnly` is the newest of those. I think `-oBatchMo... [20:08:19] (03PS1) 10Hashar: Make debian-glue voting for helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/594564 (https://phabricator.wikimedia.org/T251305) [20:09:07] (03CR) 10jerkins-bot: [V: 04-1] Make debian-glue voting for helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/594564 (https://phabricator.wikimedia.org/T251305) (owner: 10Hashar) [20:09:45] (03PS2) 10Hashar: Make debian-glue voting for helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/594564 (https://phabricator.wikimedia.org/T251305) [20:10:26] (03CR) 10jerkins-bot: [V: 04-1] Make debian-glue voting for helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/594564 (https://phabricator.wikimedia.org/T251305) (owner: 10Hashar) [20:11:02] i am rusty [20:11:56] (03PS3) 10Hashar: Make debian-glue voting for helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/594564 (https://phabricator.wikimedia.org/T251305) [20:13:49] (03CR) 10Hashar: [C: 03+2] Make debian-glue voting for helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/594564 (https://phabricator.wikimedia.org/T251305) (owner: 10Hashar) [20:14:42] (03Merged) 10jenkins-bot: Make debian-glue voting for helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/594564 (https://phabricator.wikimedia.org/T251305) (owner: 10Hashar) [20:16:51] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban): Add PushNotifications extension to release branch cut - https://phabricator.wikimedia.org/T251442 (10Mholloway) [20:17:38] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban): Add PushNotifications extension to release branch cut - https://phabricator.wikimedia.org/T251442 (10Mholloway) 05Open→03Declined We're not launching a new extension. N... [20:21:01] (03PS2) 10Jforrester: jjb: Upgrade the world to quibble 0.0.43 [integration/config] - 10https://gerrit.wikimedia.org/r/594522 [20:21:18] (03PS5) 10Jforrester: jjb: [parsoidsvc-parsertests-docker] Update path to parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/588026 (https://phabricator.wikimedia.org/T248726) (owner: 10C. Scott Ananian) [20:21:47] (03CR) 10Jforrester: [C: 03+2] "Deployed; runs in /workspace/src/services/parsoid/tests/ as expected." [integration/config] - 10https://gerrit.wikimedia.org/r/588026 (https://phabricator.wikimedia.org/T248726) (owner: 10C. Scott Ananian) [20:21:49] reloaded zuul [20:22:22] hashar: BTW I'm working through https://gerrit.wikimedia.org/r/c/integration/config/+/594522/ so no jjb changes, please. :-) [20:23:30] !log Bumped mediawiki-quibble-vendor-mysql-php72-docker to quibble 0.0.43 to test. [20:23:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:24:03] (03PS1) 10Hashar: Remove debian-glue-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/594566 (https://phabricator.wikimedia.org/T247496) [20:25:32] (03CR) 10Hashar: "Building the Debian package for Jessie doesn't do any harm for CI right now, but eventually we will have to drop support for it." [integration/config] - 10https://gerrit.wikimedia.org/r/594566 (https://phabricator.wikimedia.org/T247496) (owner: 10Hashar) [20:26:09] James_F: ;)) [20:30:58] (03PS6) 10Krinkle: [WIP] Draft Jenkins job for running tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [20:31:56] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Draft Jenkins job for running tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) (owner: 10Krinkle) [20:33:03] !log jjb: Bumped wmf-quibble-vendor-mysql-php72-docker to quibble 0.0.43 to test. [20:33:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:33:47] 10Phabricator: Phab feature request: Enable Milestones to nest within Milestones - https://phabricator.wikimedia.org/T251951 (10MBinder_WMF) [20:35:38] 10Phabricator (Upstream), 10Upstream: Phab feature request: Enable Milestones to nest within Milestones - https://phabricator.wikimedia.org/T251951 (10Peachey88) @MBinder_WMF Can this please be suggested [[ https://discourse.phabricator-community.org/ | upstream ]]? [20:37:28] Reedy: I’ve passed some info on Joaquin (the LTA) from yesterday onto T&S. Found some more digging but they have more time and skill to deal with him than me. [20:47:46] Eurgh. FileImporter's Hamcrest function is not working with Quibble 0.0.43. [20:48:27] aehbhhb [20:48:28] ;( [20:48:36] Yeh. [20:48:41] what is wrong with it? [20:49:07] (03CR) 10Jforrester: "FileImporter's Hamcrest function is not working with Quibble 0.0.43 somehow?" [integration/config] - 10https://gerrit.wikimedia.org/r/594522 (owner: 10Jforrester) [20:49:21] It's a terrible system and WMDE shouldn't use it. [20:49:38] But it shouldn't just magically disappear. If it's gone, what else has vanished? [20:49:57] I love googling stuff [20:49:58] >Hamcrest is a framework that assists writing software tests in the Java programming language [20:50:06] "Why are they using Java test stuff in PHP?: [20:50:11] They've got a PHP version. [20:50:34] Because the idioms of a (non-idiomatic) Java testing framework are /obviously/ what you should use in PHP testing. ;-) [20:50:41] Ah wikipedia, 5th sentence [20:51:13] if you get a short reproducible case, it would then be possible to bisect in quibble [20:51:26] I haaven't really carefully reviewd the last few changes that got merged [20:51:48] 7bfef6e5bdae1f3ba1f82bbc592f4eb7c8059fda Wipe repo with non-git commands [20:51:52] that one looks super fragile [20:52:26] Yeah. [20:52:34] But the behaviour it's fixing is worse. [20:52:36] * James_F sighs. [20:52:43] Maybe we'll have to revert everything. [20:53:45] (03PS7) 10Krinkle: fresh.git: Create Jenkins job for running tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [20:54:44] (03CR) 10jerkins-bot: [V: 04-1] fresh.git: Create Jenkins job for running tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) (owner: 10Krinkle) [20:54:48] (03PS8) 10Krinkle: Create a Jenkins job for Fresh that runs tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [20:55:13] James_F: then it is surely easily to track down [20:55:19] git bisect does magic sometime ;) [20:55:44] (03CR) 10jerkins-bot: [V: 04-1] Create a Jenkins job for Fresh that runs tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) (owner: 10Krinkle) [20:56:18] 10Phabricator (Upstream), 10Upstream: Phab feature request: Enable Milestones to nest within Milestones - https://phabricator.wikimedia.org/T251951 (10Aklapper) >>! In T251951#6110643, @Peachey88 wrote: > @MBinder_WMF Can this please be suggested [[ https://discourse.phabricator-community.org/ | upstream ]]?... [20:56:23] (03CR) 10Krinkle: "This is now ready for review, I've simulated a build at:" [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) (owner: 10Krinkle) [20:57:27] 10Phabricator (Upstream), 10Upstream: Phab feature request: Enable Milestones to nest within Milestones - https://phabricator.wikimedia.org/T251951 (10Aklapper) Regarding milestones, https://secure.phabricator.com/book/phabricator/article/projects/#milestones currently states that "Milestones can not have subp... [20:59:27] (03PS9) 10Krinkle: Create a Jenkins job for Fresh that runs tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [21:00:19] (03CR) 10jerkins-bot: [V: 04-1] Create a Jenkins job for Fresh that runs tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) (owner: 10Krinkle) [21:00:43] (03PS10) 10Krinkle: Create a Jenkins job for Fresh that runs tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [21:01:30] (03CR) 10jerkins-bot: [V: 04-1] Create a Jenkins job for Fresh that runs tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) (owner: 10Krinkle) [21:01:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:48] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92779 bytes in 0.974 second response time [21:14:44] (03PS11) 10Krinkle: Create a Jenkins job for Fresh that runs tests inside a Qemu VM [integration/config] - 10https://gerrit.wikimedia.org/r/593034 (https://phabricator.wikimedia.org/T250808) [21:20:11] 10Continuous-Integration-Config: Make test vote Verified+1 instead of +2 to avoid unintentional submit - https://phabricator.wikimedia.org/T226123 (10hashar) [21:20:15] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) Are {T251952} and {T251950} blockers / should they be? [21:21:24] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10brennen) I'm not rolling back at present, but yeah, marking those as blockers for going forward. [21:21:52] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10brennen) [21:22:41] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10brennen) [21:22:54] 10Scap, 10User-brennen: Investigate scap cluster_ssh idling until pressing ENTER repeatedly - https://phabricator.wikimedia.org/T223287 (10thcipriani) put together a couple of test scripts: ` #!/usr/bin/env bash SSH_AUTH_SOCK=/run/keyholder/proxy.sock /usr/bin/ssh -oBatchMode=yes -oSetupTimeout=10 -oIdentiti... [21:23:24] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) >>! In T249963#6110751, @brennen wrote: > I'm not rolling back at present, but yeah, marking those as blocke... [21:25:19] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10brennen) > ...so I caused three blockers for the train? Wow. Anyway, patches pending for both tasks, unable to test loc... [21:35:19] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) >>! In T249963#6110777, @brennen wrote: >> ...so I caused three blockers for the train? Wow. Anyway, patches... [21:38:59] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-DannyS712, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) [21:39:21] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-DannyS712, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) I should be available to fix any more issues discovered with my revision work :) [21:56:52] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-DannyS712, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) [22:03:50] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-DannyS712, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) [22:30:02] 10Phabricator (Upstream), 10Upstream: Phab feature request: Enable Milestones to nest within Milestones - https://phabricator.wikimedia.org/T251951 (10MBinder_WMF) Thanks, both. I don't believe this is a request to show subprojects on a parent workboard (as far as I can tell, that is exactly what a Milestone i... [22:39:45] I think that all the phan tests for extensions are broken right now [22:40:29] That sounds like fun [22:40:44] James_F: ^ quibble related? [22:42:35] all seem to be failing with something like https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/50766/console [22:43:08] Yeah [22:43:12] It sounds liek it [22:43:12] 23:36:37 INFO:quibble.commands:<<< Finish: Remove directory /workspace/src/extensions/MobileFrontend and any empty parents until /workspace/src, in 0.041 s [22:43:29] [21:51:48] 7bfef6e5bdae1f3ba1f82bbc592f4eb7c8059fda Wipe repo with non-git commands [22:43:29] [21:51:52] that one looks super fragile [22:43:48] uh, the line before it [22:43:48] 23:36:37 rmdir: failed to remove '/workspace/src': Device or resource busy [22:44:43] doesn't depend on the extension, I'm seing the same issues for other extensions [22:45:35] https://github.com/wikimedia/quibble/commit/7bfef6e5bdae1f3ba1f82bbc592f4eb7c8059fda [22:46:15] Pchelolo: "Now"? [22:46:29] I reverted a change to jobs two hours ago. [22:47:01] James_F: I've submitted my changes like 3 minutes ago [22:47:11] Very odd. [22:47:23] https://integration.wikimedia.org/zuul/ - [22:47:31] look at all the in flight extension changes [22:47:35] all failing [22:47:35] Why is is running the new thing. [22:47:45] Did someone fuck up? [22:48:32] !log jjb: Updating all jobs to master. [22:48:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:48:50] INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 0 [22:48:53] Hmm. [22:50:44] This is very odd. [22:50:55] Could it be contint1001/2001 fighting each other? [22:52:13] Ooooh. [22:52:14] Gah. [22:52:15] :latest [22:52:16] WTF. [22:52:18] Who did this? [22:52:19] Gah. [22:53:36] I didn't do it, I just reported a bug... don't kill the messenger :) [22:53:42] James_F: how can I help? [22:53:50] what did Pchelolo do? [22:53:56] :) [22:54:20] thcipriani: No idea. quibble-stretch-php72 on jenkins is configured different from master, and `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ 'mwext-php72-phan-docker'` doesn't push it back to master. [22:55:24] Specifically, using quibble-stretch-php72:0.0.43 instead of quibble-stretch-php72:0.0.41-s2. [22:55:31] (New quibble is broken, sadly.) [22:55:47] Pchelolo: I've manually fixed the phan job. Was it just that one or were other jobs also failing? [22:55:57] It's funny, this indirectly shows why we pin composer dependancies :) [22:56:07] Indeed. [22:56:14] Except we /do/ pin jjb versions. [22:56:24] And yet it's ignoring the command to update the job. [22:56:40] (Manually fixing to be correct and then trying to update also doesn't update the job.) [22:57:01] FWIW https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/configure is definitely using 41-s2 [22:57:18] configuring via the UI rather than jjb [22:57:24] It is now, I manually configured via the UI it. [22:57:27] ah [22:57:33] that was my only plan [22:57:37] Yeah. :-( [22:58:02] I'm half minded to do something trivial like rename "Docker" to "DockerNode" and update the world and see what's left. [22:58:13] But maybe I should stop for the day. [22:58:40] heh, that seems like something to do with more time left in the day :) [22:59:08] or something to try, rather [22:59:17] * James_F nods. [23:01:25] greping on contin1001 there are still quite a few using docker-registry.wikimedia.org/releng/quibble-stretch-php70:0.0.43 [23:01:29] *quite a few jobs [23:01:38] Hmm, that's not great. [23:01:51] Is jjb somehow updating contint2001 instead? [23:02:00] I know hashar was working on the migration. [23:02:26] that's possible [23:03:02] Is there a way to push to both? [23:03:03] grep -c 'quibble-stretch-php.*0.0.43' */config.xml | grep -v 0 | wc -l => 76 [23:03:37] James_F: the phan-seccheck too [23:03:45] Whereas my local build has 0. [23:04:15] hrm, fabfile seems to indicate it's updating contin1001 [23:04:19] Yeah. [23:04:23] But it isn't? [23:04:34] er, wait, sorry, it'd be the ini file wouldn't it [23:05:09] for jjb, that just, I guess, looks at domain [23:05:16] Pchelolo: I've manually updated that one too. [23:05:29] are you just trying to manually deploy all of the jenkins jobs from master? [23:05:35] I could give it a shot [23:05:43] thcipriani: Yeah. [23:06:14] thcipriani: `tox -e jenkins-jobs -- --conf jenkins_jobs.ini update ./jjb/ '*'` should update the world. [23:06:26] Make sure you're on master first. [23:06:55] (03CR) 10Jforrester: [C: 04-2] jjb: Upgrade the world to quibble 0.0.43 [integration/config] - 10https://gerrit.wikimedia.org/r/594522 (owner: 10Jforrester) [23:07:38] uhh, it's doing a lot of stuff [23:07:50] > INFO:jenkins_jobs.builder:Number of jobs generated: 425 [23:07:59] Yes, that's right. [23:08:08] The important line is how many it says it's updating. [23:08:23] I get `INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 0` [23:08:24] Hi! Is this channel where I am supposed to be for the patch I listed for deployment at this SWAT window? [23:08:30] huji: No. [23:08:36] huji: See #wikimedia-operations. [23:08:55] Thanks! [23:09:06] I get a whole bunch of "INFO:jenkins_jobs.builder:Reconfiguring jenkins job 3d2png-npm-node-6-docker" [23:09:32] Huh. [23:09:39] Are you on master? [23:09:51] yep [23:09:57] Very odd. [23:10:09] on master, no local changes [23:10:24] I *guess* this should be ok to let run [23:10:35] I guess we'll find out if I blow away all jobs [23:10:53] (03PS1) 10Jforrester: Revert "Wipe repo with non-git commands" [integration/quibble] - 10https://gerrit.wikimedia.org/r/594595 [23:10:55] (03PS1) 10Jforrester: Revert "Clone only the target project at first" [integration/quibble] - 10https://gerrit.wikimedia.org/r/594596 [23:10:57] (03PS1) 10Jforrester: Revert "Provide GitClean as a command" [integration/quibble] - 10https://gerrit.wikimedia.org/r/594597 [23:11:06] It should run in a few seconds. Is it still running for you? [23:11:17] yep [23:11:22] Huh. That's… odd. [23:11:30] ssh-agent set locally etc.? [23:11:55] I don't even know how it'd be slow nowadays. [23:12:03] Everything's meant to be encapsulated inside tox. [23:12:25] running: tox -e jenkins-jobs -- --conf ../jenkins-job-builder/etc/jenkins_jobs.ini update ./jjb [23:12:36] which is what I've always run, except usually with a job name [23:12:56] Oh, that's the old external JJB config. [23:12:57] We killed that. [23:13:12] Use integration/config.git's jenkins_jobs.ini [23:13:35] Huh. That doesn't exist. [23:13:46] I guess it's a local file. [23:14:50] I don't even have a jjb checkout locally any more. [23:17:25] grep -c 'quibble-stretch-php.*0.0.43' */config.xml | grep -v 0 | wc -l => 0 [23:17:45] Aha, good. [23:17:55] grep -c 'quibble-stretch-php.*0.0.41-s2' */config.xml | grep -v 0 | wc -l => 118 [23:18:13] so hopefully I didn't break anything new in the process of deploying :) [23:19:13] grep -c ':latest' */config.xml | grep -v 0 | wc -l => 312 [23:19:29] dunno when :latest started becoming a thing [23:22:39] It's just in the macros for a couple of builders using wikimedia-stretch: [23:24:06] thcipriani: `grep -c ':latest' output/* | grep -v ':0$' | wc -l` gets me 360 matches. [23:24:54] E.g. wmf-utils-tox-docker has two of them for wikimedia-stretch:latest. [23:25:26] grep ':latest' */config.xml | grep -v docker-registry.wikimedia.org/wikimedia-stretch:latest | wc -l => 0 [23:25:31] so it seems like it's all that image [23:25:35] Yeah. [23:25:44] Which aligns with what's in master. [23:25:47] which makes sense [23:25:52] So yay, but also we should stop using that. [23:26:02] Risks things breaking if SRE change the image. [23:27:40] alternatively you could argue that you stay up-to-date when sre make a security change for the image :P [23:27:58] Given we use it to copy things around the place… ;-D [23:28:21] but, I guess, that's probably not true unless we run "pull before we run "run" [23:28:22] Everything else uses ci-stretch (which builds off wikimedia-stretch); maybe we should switch over. [23:28:26] Yup. [23:28:49] Well, docker run will implicitly check the registry for what latest is, right? [23:29:32] I don't know that it does if the tag exists locally [23:30:02] if it doesn't have a ":latest" image locally, then it'll pull from the registry [23:30:02] It won't be local inside the agents. Or is this running directly on contint itself? [23:30:05] * James_F peers. [23:30:50] Oh, right. [23:31:03] Anyway. I'm going to stop working now. Clean-up for later. [23:32:12] fr.ex. integration-agent-docker-[1001-1004,1007-1008,1010,1013,1016].integration.eqiad.wmflabs have a different :latest than integration-agent-docker-[1005-1006,1009,1011-1012].integration.eqiad.wmflabs according to cumin :P