[00:00:49] hmm [00:05:33] Anyway I copied the eventlogging and deploy_service (aka servicedeploy?) keys into place, restarted keyholder-proxy and armed it [00:06:08] don't know where the analytics and dumpsdeploy keys live but their passphrases are in the usual location [00:06:24] phabricator one is not [00:06:27] going to sleep [00:07:12] (maybe the analytics and dumpsdeploy keys are simply in the location puppet expects to find them and not files/ssh/tin...) [03:16:57] (03PS1) 10MaxSem: FunctionAnnotationsSniff: whitelist @before [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/544090 [05:56:18] (03CR) 10Daimona Eaytoy: [C: 03+2] FunctionAnnotationsSniff: whitelist @before [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/544090 (owner: 10MaxSem) [05:56:48] (03Merged) 10jenkins-bot: FunctionAnnotationsSniff: whitelist @before [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/544090 (owner: 10MaxSem) [07:15:08] (03PS2) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 [07:15:33] (03CR) 10jerkins-bot: [V: 04-1] Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4) [07:17:14] (03CR) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches (033 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4) [08:00:03] Project mwcore-phpunit-coverage-master build #243: 04FAILURE in 5 hr 0 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/243/ [09:07:48] (03PS3) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 [09:08:12] (03CR) 10jerkins-bot: [V: 04-1] Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4) [09:09:52] (03PS4) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 [09:11:52] (03PS5) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 [10:27:04] RECOVERY - Puppet errors on integration-agent-stretch-1001 is OK: OK: Less than 1.00% above the threshold [2.0] [12:02:35] hashar: 👋 I have a question about quibble in relation to the tools/api-testing repo. [12:02:55] chatchat: invalid unicode sequence [12:03:04] * hashar fixes for emoji support [12:03:08] clarakosi: sure! :) [12:03:22] hahaha [12:03:24] I want to write integration tests against the experimental REST API. In order to do so I have to enable the endpoints. [12:03:38] Its a simple `$wgEnableRestAPI = true;` Do I add that to quibble’s local_settings.php or is there another way to enable that? [12:03:55] ahhhh [12:03:56] :-\ [12:03:57] no [12:04:17] so quibble really just does: php maintenance/install.php --what --ever --else [12:04:41] then indede there is builtin local_settings.php file that has: some setting ... include LocalSettings.php ... some more settings [12:04:51] with now good way to inject any additional settings [12:05:06] so the only chance right now is to tweak that local_settings.php file [12:05:19] but [12:05:20] then [12:05:44] that means the setting would be used by any other jobs that use quibble [12:06:12] hmmm is there a way to have our own localsetting file for the repo? [12:06:54] I think we have/had it for selenium tests, but I think that got removed [12:08:15] that the same issue Daniel had to have $wgSecretKey set to a specific value [12:08:31] or that awight had to set $wgEnableUploads = true ; for an integration test involving uploading files [12:08:39] which for now has to be hardcoded in local_settings.php [12:09:09] the pity is that since that is hardcoded in Quibble code, that means any addition of settings : A) affects everyone / all jobs B) requires a new release of quibble [12:09:13] which don't scale clearly [12:09:31] so I guess we should have a feature to be able to inject / require an arbitrary file [12:10:11] I thought about having that supported by mediawiki maintenance/install.php [12:10:29] but we cant just add an option to that script for each of the $wg setting we want to potentially vary from the default [12:11:03] imagine maintenance/install.php --secret-key foo --enable-uploads --enable-Rest-API [12:11:53] clarakosi: sorry I am brain dumping my thoughts [12:11:58] ohh that would be helpful but like you said we'd need to add each possible option [12:12:03] no worries :) [12:12:04] yeah [12:12:06] so [12:12:21] awight last week made a fix to Quibble local_settings.php [12:12:40] so that we can have settings BEFORE LocalSettings.php and AFTER LocalSettings.php (to override autogenerated settings) [12:12:53] but that clearly does not scale since you now as for $wgEnableRestAPI [12:12:59] and surely others would ask for more setting tweaks [12:13:26] so I am wondering wether we could make maintenance/install.php to accept some --pre / --after parameters which would take a php file as argument [12:13:44] and when LocalSettings is generated, those files are included at the top and at the end of the generated file [12:14:02] then the CI job, or a local run could provide files that have any arbitrary settings [12:14:53] that would be great for testing other settings [12:14:55] and for the api-testing CI job, we could then set it to prepend/append files that are in the api-testing git repo and thus under full control of developers [12:15:10] saving the hassle of going through Quibble developers / cutting a new release / rebuilding the CI containers etc [12:15:12] yeah [12:16:36] should I create a ticket for that? [12:16:51] clarakosi: yes please! :-\ [12:17:01] ok! Will do [12:17:02] :-\ <-- because I wish we had an immediate solution [12:17:32] oh and api-testing is not an extension bah [12:17:49] so you can even use a magic hack such as a using a SetupAfterHook or onregistration function to tweak the setting you need :/ [12:18:22] which is what folks do on extensions, so that the extension can potentially inject different setting when it detects it is run under CI [12:18:56] if ( isset( $wgWikimediaJenkinsCI ) && $wgWikimediaJenkinsCI ) { /** hack settings **/ } [12:19:04] that $wg setting is in Quibble local_settings.ph [12:19:06] p [12:19:54] ohhh so that's what that was about [12:20:21] though most extensions just have "sane" settings [12:20:54] a list of those hacks are https://codesearch.wmflabs.org/search/?q=wgWikimediaJenkinsCI&i=nope&files=&repos= [12:21:09] but anyway, that does not apply to mediawiki/tools/api-testing [12:21:17] or hmm [12:21:24] [12:21:38] potentially we could have a mediawiki/extensions/ApiTestingSettings dummy extension [12:22:03] which quibble would clone ( just pass it as an argument ) and the installer would then detect that extension (since Quibble runs maintenance/install.php --with-extensions ) [12:22:09] so that the generated LocalSettings.php file would have: [12:22:18] wfLoadExtension( 'ApiTestingSettings' ); [12:22:31] which then in turns lets developers do dirty settings hack into that regular mediawiki extension [12:22:48] such as changing settings via some hook or black magic you mediawiki developers might have available to you [12:23:20] I think I previously used setupAfterCache hook, $wgExtensionFunctions[] ( I think that is deprecated now) or extension.json "onRegistration" (which I think is not recommended) [12:23:30] but that extra extension might be too hacky [12:23:57] so as usually the problem is easy: have the Quibble test runner to inject arbitrary settings before/after the generated config [12:24:08] but there are multiple potential solutions to the problem :\ [12:26:03] hmmm the extension would probably get us unblocked the quickest but I'm not sure if this is a problem unique to us or if other people are constantly asking to update quibble's local_settings [12:30:06] clarakosi: the only two recent use cases were awight to set $wgEnableUploads, that is for a selenium integration tests to ensure uploading file via the browser result in whatever [12:30:31] and the second case was forcing $wgSecretKey for api-testing, in order to be able to trigger Special:RunJobs [12:30:45] there are other use cases for selenium tests I believe [12:30:52] but yeah overall, that is a rare demand [12:31:32] I think I am worrying about adding more madness to Quibble, and get to extract the logic out of Quibble so that developers have full control [12:32:18] (also isn't RestBase the rest api?) [12:33:45] clarakosi: $wgEnableRestAPI is just a feature flag to enable /rest.php entry point isn't it ? [12:33:54] it probably does not have much other impacts beside enabling that [12:34:12] yup its just to enable that endpoint [12:35:41] so yeah probably going to be harmless ;D [12:37:28] wait so does that mean we can just add it or should we still consider allowing repos to have their own localsettings [12:37:32] or both? [12:40:07] (03PS1) 10Reedy: Add GlobalCheckUser to CI [integration/config] - 10https://gerrit.wikimedia.org/r/544186 [12:40:28] (03CR) 10Reedy: [C: 03+2] Add GlobalCheckUser to CI [integration/config] - 10https://gerrit.wikimedia.org/r/544186 (owner: 10Reedy) [12:42:30] (03Merged) 10jenkins-bot: Add GlobalCheckUser to CI [integration/config] - 10https://gerrit.wikimedia.org/r/544186 (owner: 10Reedy) [12:43:00] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/544186 [12:43:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:45:36] clarakosi: so for that specific setting $wgEnableRestAPI, given it just enables /rest.php we should just add it to local_settings.php [12:45:50] with all of the text I wrote before being me over thinking for different use cases [12:46:34] clarakosi: so essentially, yes lets fill a task. The patch for Quibble is trivial : just add to local_settings.php [12:46:43] ok got it [12:48:07] clarakosi: sorry, really I just brain dumped my thoughts as I had them [12:48:37] or how it takes me ~ 25 minutes just to figure out that yeah the fix is just as easy than adding to local_settings.php bah [12:48:42] no worries I prefer to see the thinking process :) [12:49:13] thanks :] [12:56:47] (03PS1) 10Clarakosi: Enable REST API for testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/544189 (https://phabricator.wikimedia.org/T235564) [12:59:33] hashar: Alright I've added you to both the ticket (T235865) and patch listed above 👆 [12:59:33] T235865: Modify Quibble to accept a LocalSettings.php file - https://phabricator.wikimedia.org/T235865 [12:59:51] clarakosi: yeah. Devil thing, should that inject before or after extensions get loaded ? :D [13:00:50] not that it matters in this case anywya [13:00:54] wait does the order matter [13:01:31] now I want to move it [13:03:49] so yeah [13:04:06] after LocalSettings mean the setting applies AFTER all the wfLoadExtensions() [13:04:15] and it also mean that overrides stuff magically generated by install.php [13:05:08] wait now I'm confused [13:05:38] sorry :-\ [13:06:30] if we put it before the extensions wouldn't that mean the extensions can override it [13:06:49] and by extensions are we talking about `$wgWikimediaJenkinsCI`? [13:08:43] $wgWikimediaJenkinsCI is just a custom setting injected by Quibble [13:08:46] which is a hint to extension [13:08:56] it is really a hack / legacy setting we have introduced years and years ago [13:09:15] for some case when extensions need a different default behavior when running under CI [13:09:23] eg to use a slightly different config than the default one [13:11:34] (03CR) 10Hashar: Enable REST API for testing (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/544189 (https://phabricator.wikimedia.org/T235564) (owner: 10Clarakosi) [13:12:08] clarakosi: the patch you did just mean that $wgEnableRestAPI would still be false when wfLoadExtension() is called [13:12:17] but that is probably not an issue [13:12:55] ohhh [13:12:56] anyway, I think it is preferable to move $wgEnableRestAPI = true; above the include(LocalSettings-installer.php); [13:12:58] ok [13:13:07] Alright I'll move it.. one se [13:13:10] so that it is available to wfLoadExtenion() calls [13:13:11] sec* [13:13:15] though probably it does not matter [13:13:37] I don't really know what kind of code is executed when invoking wfLoadExtension() [13:15:11] (03PS2) 10Clarakosi: Enable REST API for testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/544189 (https://phabricator.wikimedia.org/T235564) [13:17:58] clarakosi: sorry I did not have any good rationale / proof to back up my claim to move the setting above :D [13:18:28] (03CR) 10Hashar: [C: 03+2] Enable REST API for testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/544189 (https://phabricator.wikimedia.org/T235564) (owner: 10Clarakosi) [13:18:43] you eventually did though :) [13:18:47] clarakosi: I will cut a new release / rebuild a ci container and update the CI job [13:18:59] hashar: thanks!! [13:19:05] or maybe there is another change I can get merged :D [13:19:31] (03Merged) 10jenkins-bot: Enable REST API for testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/544189 (https://phabricator.wikimedia.org/T235564) (owner: 10Clarakosi) [13:20:15] * hashar reads releasing doc [13:28:56] (03PS1) 10Hashar: Release Quibble 0.0.39 [integration/quibble] - 10https://gerrit.wikimedia.org/r/544192 (https://phabricator.wikimedia.org/T235564) [13:31:54] 14:31:29 [FATAL tini (7)] exec /srv/composer/composer-install-dev-only failed: Permission denied [13:32:17] what? [13:32:39] https://gerrit.wikimedia.org/r/544193 [13:32:46] https://integration.wikimedia.org/ci/job/operations-mw-config-php72-composer-lint-docker/5376/console [13:33:24] dockerfiles/composer/Dockerfile.template:COPY composer-install-dev-only.bash /srv/composer/composer-install-dev-only [13:33:30] I guess the container is broken [13:34:00] and missing the executable bit somehow? :\ [13:35:11] so yeah [13:35:19] releng/composer has the executable bit [13:35:32] the one from releng/composer-php70 does not [13:37:15] (03CR) 10Hashar: [C: 03+2] Release Quibble 0.0.39 [integration/quibble] - 10https://gerrit.wikimedia.org/r/544192 (https://phabricator.wikimedia.org/T235564) (owner: 10Hashar) [13:38:02] (03Merged) 10jenkins-bot: Release Quibble 0.0.39 [integration/quibble] - 10https://gerrit.wikimedia.org/r/544192 (https://phabricator.wikimedia.org/T235564) (owner: 10Hashar) [13:44:15] !log Tag Quibble 0.0.39 @ a1e0b85 # T235023 T235564 [13:44:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:44:19] T235023: Selenium test "User should be able to log in @daily" fails with Quibble 0.0.37 - https://phabricator.wikimedia.org/T235023 [13:44:19] T235564: Implement Integration Tests for Existing iOS Endpoints - https://phabricator.wikimedia.org/T235564 [13:45:29] (03PS1) 10Hashar: changelog: begin new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/544194 [13:49:08] (03PS1) 10Hashar: docker: bump Quibble to 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544195 (https://phabricator.wikimedia.org/T235564) [13:51:15] (03PS1) 10Hashar: Switch api-testing job to Quibble 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544196 (https://phabricator.wikimedia.org/T235564) [13:51:25] clarakosi: ^^^ madness required to update CI with a new version of Quibble :D [13:51:49] !log Building Quibble 0.0.39 CI containers [13:51:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:52:43] (03CR) 10Hashar: [C: 03+2] docker: bump Quibble to 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544195 (https://phabricator.wikimedia.org/T235564) (owner: 10Hashar) [13:54:44] (03Merged) 10jenkins-bot: docker: bump Quibble to 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544195 (https://phabricator.wikimedia.org/T235564) (owner: 10Hashar) [13:59:01] Ohhh no... yeah we need a better solution for next time [14:08:00] clarakosi: yeah eventually : ] [14:08:10] hence my wall of text earlier [14:10:10] (03PS1) 10Hashar: docker: update quibble-stretch-bundle dependency [integration/config] - 10https://gerrit.wikimedia.org/r/544200 [14:18:21] yeah :( [14:18:53] The change shouldn't be that hard so I'll see if we can allocate time for it next sprint [14:31:25] clarakosi: job should be updated now [14:31:35] clarakosi: took longer since I am multi tasking on other stuff as well [14:32:10] hashar: already? thanks! [14:32:21] rechecked https://gerrit.wikimedia.org/r/#/c/mediawiki/tools/api-testing/+/543953/ [14:32:52] thanks :) [14:33:19] (03CR) 10Hashar: [C: 03+2] "Jobs updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/544196 (https://phabricator.wikimedia.org/T235564) (owner: 10Hashar) [14:33:25] (03CR) 10Hashar: [C: 03+2] docker: update quibble-stretch-bundle dependency [integration/config] - 10https://gerrit.wikimedia.org/r/544200 (owner: 10Hashar) [14:35:44] (03Merged) 10jenkins-bot: Switch api-testing job to Quibble 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544196 (https://phabricator.wikimedia.org/T235564) (owner: 10Hashar) [14:35:48] (03Merged) 10jenkins-bot: docker: update quibble-stretch-bundle dependency [integration/config] - 10https://gerrit.wikimedia.org/r/544200 (owner: 10Hashar) [14:37:50] clarakosi: and tentatively you should be able to run quibble on your local machine and have the same result as on CI [14:38:10] delta the composer/php versions on your environment :D [14:38:35] looks like the suite managed to hit GET /page/{title}/history/counts/edits [14:38:41] Good to know [14:39:09] yeah I forgot to count undo as an edit 🤦‍♀️ [15:28:23] 10Release-Engineering-Team (CI & Testing services), 10Cloud-VPS, 10Tools, 10Wikimedia-Incident, 10cloud-services-team (Kanban): Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07 - https://phabricator.wikimedia.org/T234834 (10Andrew) 05Open→03Resolved As far a... [15:31:12] Doint you just hate it when you hit traffic in London... [15:33:46] Don't drive in London ;P [15:41:09] Are there known issues with CI right now? [15:42:26] (btw, congrats on promotions thcipriani and greg-g — both well deserved!) [15:42:50] Anything specifically? [15:43:04] andrewbogott: mw-config jobs are borked (with a missing execute bit thing) if that's what you mean [15:43:11] That's what I mean :) [15:43:12] thanks [15:45:25] andrewbogott: thanks :) [15:45:35] andrewbogott: thank you! [15:46:24] also "missing execute bit"? [15:46:43] thcipriani: for example https://integration.wikimedia.org/ci/job/operations-mw-config-php72-composer-lint-docker/5381/console [15:47:00] https://phabricator.wikimedia.org/T235879 [15:47:27] ah, yeah, I found a job: exec /srv/composer/composer-install-dev-only failed: Permission denied [15:48:09] [14:35:19] releng/composer has the executable bit [15:48:09] [14:35:32] the one from releng/composer-php70 does not [15:49:57] Reedy: Lol [15:49:58] I’m not the coach is :P [15:52:21] hrm, looks like composer-php70 is new as of yesterday [15:54:02] Is it actually PHP 7.0? [15:54:11] Seems a bit odd when it's not supported, we don't use it in prod... [15:55:33] kinda looks like both releng/php and releng/php70 are php7.0. The difference is that the latter installs php-intl [15:57:03] testing mw-config with php70 seems odd. Core/extensions have compatibility to think about. mwconfig: not sure why. [15:58:17] also, not sure why it's not executable unless the executable bit was removed from https://gerrit.wikimedia.org/r/p/integration/composer.git:vendor/bin/composer sometime between releng/composer and releng/composer-php70 [15:58:59] though...that doesn't look likely [16:00:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Fundraising-Backlog, 10Patch-For-Review: Create composer-test-php70 docker image for fundraising tech's crm tests - https://phabricator.wikimedia.org/T230446 (10Ejegg)... [16:01:36] oh... [16:06:32] (03PS1) 10Thcipriani: docker: composer-php70 set executable bit on scripts [integration/config] - 10https://gerrit.wikimedia.org/r/544219 (https://phabricator.wikimedia.org/T235879) [16:14:33] (03CR) 10Reedy: [C: 03+1] docker: composer-php70 set executable bit on scripts [integration/config] - 10https://gerrit.wikimedia.org/r/544219 (https://phabricator.wikimedia.org/T235879) (owner: 10Thcipriani) [16:43:41] (03PS6) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 [16:44:10] (03CR) 10jerkins-bot: [V: 04-1] Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4) [16:48:24] (03CR) 10Daimona Eaytoy: [C: 03+1] Add SocialProfile and VoteNY as phan dependencies for ImageRating [integration/config] - 10https://gerrit.wikimedia.org/r/543843 (owner: 10Jack Phoenix) [17:01:06] (03CR) 1020after4: [C: 03+2] docker: composer-php70 set executable bit on scripts [integration/config] - 10https://gerrit.wikimedia.org/r/544219 (https://phabricator.wikimedia.org/T235879) (owner: 10Thcipriani) [17:03:22] (03Merged) 10jenkins-bot: docker: composer-php70 set executable bit on scripts [integration/config] - 10https://gerrit.wikimedia.org/r/544219 (https://phabricator.wikimedia.org/T235879) (owner: 10Thcipriani) [17:09:15] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/544219 [17:09:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:23:18] (03PS1) 10Thcipriani: docker: composer-php70 rebuild all dependent images [integration/config] - 10https://gerrit.wikimedia.org/r/544230 (https://phabricator.wikimedia.org/T235879) [17:23:34] ^ James_F could you take a look at that please? I am flailing :) [17:47:53] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:47:53] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:58:53] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: exec /srv/composer/composer-install-dev-only failed: Permission denied - https://phabricator.wikimedia.org/T235879 (10Jdforrester-WMF) Hmm: ` ~/D/c/integration-config   master $…  ls -l dockerfiles/*/*.sh | sort... [17:59:57] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: exec /srv/composer/composer-install-dev-only failed: Permission denied - https://phabricator.wikimedia.org/T235879 (10Jdforrester-WMF) (Or remove the exec bit from the `example-run.sh` files for those that have it?) [18:00:01] thcipriani: Sure, will do. [18:00:44] I did the docker-pkg update thing, FWIW [18:00:46] thanks :) [18:01:01] You mean the `fab deploy_docker` one? [18:02:24] thcipriani: I already fixed the chmod for composer-test-php70 and composer-package-php70 yesterday, but missed the underlying issue – see b4c22a019e1df5232b81e3aa095136c987391dcd and 8837b38b46da9d14d1cf9903201931f54f16210f [18:03:06] But the update cascade looks right; does it not work? [18:06:35] there was another change needed this morning I34b213cf03b625d3d3177fa8df52ad31b48add3d [18:06:40] Yeah. [18:06:57] and then I did that, and realized that php70 wasn't actually used, just included :) [18:07:03] Yup. :-) [18:07:14] We almost never actually use php70. [18:07:27] We just copy the contents of /srv/composer [18:07:38] Because docker doesn't support multiple inheritance. [18:08:19] if the upgrade cascade looks right, then I guess i can do that, and then update a bunch of jobs? or just some jobs? what's typical here? [18:08:29] Update the world, sadly. [18:08:41] See my work yesterday (which caused this breakage, whoops). [18:08:55] I split the jjb update into direct and cascade images for clarity. [18:10:29] k, so procedure is merge this, rebuild all the things, and patch all the jjb [18:10:36] and then deploy those changes to jenkins jobs? [18:11:50] (03PS1) 10Jforrester: dockerfiles: Fix run.sh chmod for php70-compile and lintr [integration/config] - 10https://gerrit.wikimedia.org/r/544238 [18:11:56] Yeah, one by one and check a run. [18:12:09] Or, apparently, check multiple runs where the same jobs are used differently. Oops. [18:12:13] It's… tedious. [18:12:32] And I've found two others that need fixing ^^, so let's do the bumps all at once. [18:13:00] Though I don't know if php70-compile is actually ever used in practice, and lintr certainly isn't (analytics don't use R much any more). [18:13:52] The "easier" way is to change all the jjbs over to use :latest and play roulette each time we upgrade images. ;-) [18:13:53] 10Project-Admins: Create Project: Watchlist-Expiry - https://phabricator.wikimedia.org/T235686 (10MBinder_WMF) > #Community-Tech: Would you like [Herald Rule](https://www.mediawiki.org/wiki/Phabricator/Help/Herald_Rules) H260 to be updated to include #Expiring-Watchlist-Items ? @Aklapper Good idea, yes please [18:14:14] Damn our focus on quality/reliability, it adds so much work! :-) [18:15:04] (03PS2) 10Jforrester: dockerfiles: Fix run.sh chmod for php70-compile and lintr [integration/config] - 10https://gerrit.wikimedia.org/r/544238 (https://phabricator.wikimedia.org/T235879) [18:15:10] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Fix run.sh chmod for php70-compile and lintr [integration/config] - 10https://gerrit.wikimedia.org/r/544238 (https://phabricator.wikimedia.org/T235879) (owner: 10Jforrester) [18:15:26] (03CR) 10Jforrester: [C: 03+2] Add SocialProfile and VoteNY as phan dependencies for ImageRating [integration/config] - 10https://gerrit.wikimedia.org/r/543843 (owner: 10Jack Phoenix) [18:15:58] (03CR) 10Jforrester: [C: 04-2] "Also not ready here, this is just aspirational. ;-) See the stack below." [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) (owner: 10Jforrester) [18:16:52] (03Merged) 10jenkins-bot: dockerfiles: Fix run.sh chmod for php70-compile and lintr [integration/config] - 10https://gerrit.wikimedia.org/r/544238 (https://phabricator.wikimedia.org/T235879) (owner: 10Jforrester) [18:17:21] The lintr build is from C source, and so it epically tedious. [18:17:45] (03Merged) 10jenkins-bot: Add SocialProfile and VoteNY as phan dependencies for ImageRating [integration/config] - 10https://gerrit.wikimedia.org/r/543843 (owner: 10Jack Phoenix) [18:19:35] !log Docker: Building lintr:0.2.1 and php70-compile:0.1.4 [18:19:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:28:18] (03CR) 10Jforrester: [C: 03+2] docker: composer-php70 rebuild all dependent images [integration/config] - 10https://gerrit.wikimedia.org/r/544230 (https://phabricator.wikimedia.org/T235879) (owner: 10Thcipriani) [18:29:39] Huh. [18:29:49] thcipriani: Did you run `fab deploy_zuul` just now? [18:30:10] (03Merged) 10jenkins-bot: docker: composer-php70 rebuild all dependent images [integration/config] - 10https://gerrit.wikimedia.org/r/544230 (https://phabricator.wikimedia.org/T235879) (owner: 10Thcipriani) [18:30:58] James_F: nope [18:31:20] It's saying CI already is at the point with the SocialProfile and VoteNY phan dependencies patch [18:32:02] Odd. [18:32:09] * James_F runs a clean sync just in case. [18:32:30] !log Zuul clean sync as the last patch was seemingly already deployed? [18:32:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:32:51] James_F: hey james :) [18:33:01] Hey hashar. Was that you? :-) [18:33:16] for the deploy not sure [18:33:27] but that can be checked on contint1001 direclty by looking at the git log maybe [18:33:30] if there is something pending [18:33:38] I don't think I have done any zuul deployment today [18:33:54] Kk, will have a poke. [18:35:19] also reedy reported an issue with some script in composer that lacks an executable bit [18:35:23] and [18:35:27] Yeah, that's what I'm fixing. [18:35:34] ah great [18:35:38] I hate Docker [18:35:41] Well, thcipriani was fixing and I took over. ;-) [18:35:41] and how we gotta copy paste [18:35:51] Yeah, it's tedious. [18:36:10] I think I have seen a system that is a smarter to compose images based on various layers [18:36:10] But multiple inheritance of containers is conceptually very hard. [18:36:20] kind of cherry picking multiple layers to compose the result image [18:36:36] Hmm, interesting. [18:36:41] cause really, it is in the end just about merging part of file trees [18:37:12] so assuming that the tree of two layers do not share any files in common, you can just "cherry-pick" them [18:37:18] and forge a new layer [18:37:42] A has /etc/foo.conf , B has /etc/bar.conf [18:37:49] one could craft a C image by merging A and B [18:37:56] and C would have /etc/foo.conf and /etc/bar.conf [18:37:57] magic [18:38:42] at some point I considered to just build abunch of images and then use docker export on the ones I wanted and then compose a tarball "manually" [18:38:47] but that is not smart enough [18:39:09] anyway, I also had to cut a new Quibble 0.0.39 release to serve mediawiki team [18:41:01] Hmm, OK. [18:41:17] CI /etc/zuul/wikimedia has the patches. [18:41:51] * James_F proceeds. [18:42:34] !log Docker: Rebuilding the world onto composer-php70:0.1.1 [18:42:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:43:13] \o/ [18:43:44] I am off, gotta clean up stuff at home [18:43:51] Have fun. [18:43:53] poke on telegram if need be. I am not far :] [18:52:38] Oh, right, that means I need to bump the fresnel image. [18:52:46] * James_F tuts hashar. [18:57:14] (03PS1) 10Jforrester: dockerfiles: [quibble-fresnel] Fix numbering for release for 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544243 [18:57:16] (03PS1) 10Jforrester: jjb: Bump all non-quibble jobs to use latest images [integration/config] - 10https://gerrit.wikimedia.org/r/544244 (https://phabricator.wikimedia.org/T235879) [18:57:18] (03PS1) 10Jforrester: jjb: Bump all quibble jobs to 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544245 [18:58:05] (03CR) 10Jforrester: "Leaving this to hashar for now (unless I get really bored over the weekend)." [integration/config] - 10https://gerrit.wikimedia.org/r/544245 (owner: 10Jforrester) [18:58:20] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [quibble-fresnel] Fix numbering for release for 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544243 (owner: 10Jforrester) [18:59:03] Yippee, build fixed! [18:59:04] Project mwcore-phpunit-coverage-master build #244: 09FIXED in 3 hr 59 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/244/ [19:00:00] (03Merged) 10jenkins-bot: dockerfiles: [quibble-fresnel] Fix numbering for release for 0.0.39 [integration/config] - 10https://gerrit.wikimedia.org/r/544243 (owner: 10Jforrester) [19:00:35] * James_F twiddles thumbs waiting for docker build again. [19:01:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review: exec /srv/composer/composer-install-dev-only failed: Permission denied - https://phabricator.wikimedia.org/T235879 (10Jdforrester-WMF) p:05Triage→03High a:03Jdforrester-WMF [19:02:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review: CI broken with composer-install-dev-only and security-check scripts not executable; can't merge in mwconfig, other repos - https://phabricator.wikimedia.org/T235879 (10Jdforrester-WMF) [19:13:47] (03PS2) 10Jforrester: jjb: Bump all non-quibble jobs to use latest images [integration/config] - 10https://gerrit.wikimedia.org/r/544244 (https://phabricator.wikimedia.org/T235879) [19:18:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review: CI broken with composer-install-dev-only and security-check scripts not executable; can't merge in mwconfig, other repos - https://phabricator.wikimedia.org/T235879 (10Jdforrester-WMF) OK, I've fixed the jo... [19:18:35] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10User-DannyS712, 10ci-test-error: Jenkins failing - 'Permission denied' - https://phabricator.wikimedia.org/T235816 (10Jdforrester-WMF) [19:18:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review: CI broken with composer-install-dev-only and security-check scripts not executable; can't merge in mwconfig, other repos - https://phabricator.wikimedia.org/T235879 (10Jdforrester-WMF) [19:19:18] 10Release-Engineering-Team (Pipeline), 10Operations, 10Release Pipeline, 10serviceops, 10Goal: Self-service Deployment Pipeline - https://phabricator.wikimedia.org/T228676 (10Ladsgroup) p:05Normal→03High [19:30:10] thcipriani https://www.gerritcodereview.com/2019-10-15-esc-minutes.html#security-issue-with-ldap-and-java-11 [19:32:39] > Since Java 11 is not yet an officially supported runtime anyway, this issue doesn’t need restricted visibility. [19:32:43] fair enough [19:33:13] i doin't belive we use startTLS with ldap [19:35:02] so we shouldn't be affected i think? [19:35:16] (we use ldaps though) [19:45:02] grmbbl :-\ [19:45:07] I could not even use the proper version [19:47:32] (03CR) 10Hashar: [C: 03+1] ":]" [integration/config] - 10https://gerrit.wikimedia.org/r/540694 (owner: 10Jforrester) [19:49:17] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/544244 (https://phabricator.wikimedia.org/T235879) (owner: 10Jforrester) [19:52:15] (03Merged) 10jenkins-bot: jjb: Bump all non-quibble jobs to use latest images [integration/config] - 10https://gerrit.wikimedia.org/r/544244 (https://phabricator.wikimedia.org/T235879) (owner: 10Jforrester) [19:52:39] (03PS2) 10Jforrester: Enforce commit-message-validator for this repo [integration/config] - 10https://gerrit.wikimedia.org/r/543038 (owner: 10Hashar) [19:53:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10Patch-For-Review: CI broken with composer-install-dev-only and security-check scripts not executable; can't merge in mwconfig, other repos - https://phabricator.wikimedia.org/T235879 (10Jdforrester-WMF) 05Open→03Resolved [19:53:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Fundraising-Backlog, 10Patch-For-Review: Create composer-test-php70 docker image for fundraising tech's crm tests - https://phabricator.wikimedia.org/T230446 (10Jdforres... [19:55:44] (03CR) 10Jforrester: [C: 03+2] Enforce commit-message-validator for this repo [integration/config] - 10https://gerrit.wikimedia.org/r/543038 (owner: 10Hashar) [19:57:34] (03Merged) 10jenkins-bot: Enforce commit-message-validator for this repo [integration/config] - 10https://gerrit.wikimedia.org/r/543038 (owner: 10Hashar) [19:58:21] (03CR) 10Jforrester: docker: bump Quibble to 0.0.39 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/544195 (https://phabricator.wikimedia.org/T235564) (owner: 10Hashar) [20:07:06] RECOVERY - Puppet staleness on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:10:34] RECOVERY - Puppet staleness on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:12:59] RECOVERY - Puppet staleness on deployment-sessionstore02 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:13:28] RECOVERY - Puppet staleness on deployment-sessionstore01 is OK: OK: Less than 1.00% above the threshold [3600.0] [20:23:48] RECOVERY - Free space - all mounts on integration-agent-docker-1006 is OK: OK: All targets OK [20:34:23] !log integration: sudo cumin --force 'name:docker' 'docker container prune -f' [20:34:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:34:27] !log integration: sudo cumin --force 'name:docker' 'docker image prune -f' [20:34:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:41:37] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10Notifications, 10CPT Initiatives (Multi-DC Echo Notification Storage), 10Core Platform Team Workboards (Green): Provision deployment-prep instance of echostore - https://phabricator.wikimedia.org/T235920 (10Reedy) [20:57:28] (03CR) 10Jforrester: "To confirm, this is just a fix-up for the next round of releases so doesn't touch any changelog/cascade?" [integration/config] - 10https://gerrit.wikimedia.org/r/544200 (owner: 10Hashar) [21:03:27] (03CR) 10Hashar: [C: 03+2] "It is just to fix the dependency chain, no need to rebuild anything." [integration/config] - 10https://gerrit.wikimedia.org/r/544200 (owner: 10Hashar) [21:03:39] James_F: sorry I have made that fix in a hurry :\ [21:03:45] but yeah no need to rebuild anything [21:06:18] (03PS1) 10Ammarpad: Add link to Design Style Guide at doc homepage [integration/docroot] - 10https://gerrit.wikimedia.org/r/544257 (https://phabricator.wikimedia.org/T235715) [21:07:37] (03CR) 10Jforrester: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/544200 (owner: 10Hashar) [21:18:23] (03CR) 10Hashar: [C: 03+2] "> Yeah, I've been thinking about little CI-for-CI improvements we could make like that." [integration/config] - 10https://gerrit.wikimedia.org/r/544200 (owner: 10Hashar) [21:19:10] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34.0-rc.1 - https://phabricator.wikimedia.org/T235922 (10Reedy) [21:26:08] !log create cloud/instance-puppet repo per andrewbogott request [21:26:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:10] * hashar vanishes [21:34:32] (03PS1) 10Thcipriani: deployments-calendar: Fix American train output [tools/release] - 10https://gerrit.wikimedia.org/r/544261 [21:35:12] (03CR) 10Thcipriani: [C: 03+2] deployments-calendar: Fix American train output [tools/release] - 10https://gerrit.wikimedia.org/r/544261 (owner: 10Thcipriani) [21:35:49] (03Merged) 10jenkins-bot: deployments-calendar: Fix American train output [tools/release] - 10https://gerrit.wikimedia.org/r/544261 (owner: 10Thcipriani) [21:48:50] 10MediaWiki-Releasing, 10Core Platform Team, 10MW-1.34-notes, 10MW-1.34-release: Release MW 1.34.0-rc.1 - https://phabricator.wikimedia.org/T235922 (10Reedy) [22:23:47] (03CR) 10Dduvall: Fix up branch.py so that it's suitable for wmf/ production branches (033 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4) [22:36:07] (03CR) 1020after4: "Thanks for the review, DDuvall! Responses inline." (033 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4) [22:40:48] (03CR) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/543248 (owner: 1020after4) [23:20:59] Project beta-update-databases-eqiad build #37502: 04FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/37502/ [23:22:39] timeout reading from etcd