[00:22:02] 10Continuous-Integration-Infrastructure (Slipway), 10Wikidata, 10Wikidata Query UI: Migrate wikidata-query-gui-build to Docker containers - https://phabricator.wikimedia.org/T210286 (10Addshore) [00:33:22] PROBLEM - Free space - all mounts on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out) [00:33:24] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: (Service Check Timed Out) [00:35:15] PROBLEM - Free space - all mounts on deployment-mx02 is CRITICAL: (Service Check Timed Out) [00:35:15] PROBLEM - Puppet staleness on integration-r-lang-01 is CRITICAL: (Service Check Timed Out) [00:35:15] PROBLEM - Free space - all mounts on deployment-dumps-puppetmaster02 is CRITICAL: (Service Check Timed Out) [00:35:15] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: (Service Check Timed Out) [00:35:15] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: (Service Check Timed Out) [00:36:13] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: (Service Check Timed Out) [03:13:32] (03CR) 1020after4: [C: 03+2] Some bug fixes [releng/local-charts] - 10https://gerrit.wikimedia.org/r/517780 (owner: 10Jeena Huneidi) [03:46:03] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T220736 (10Jdforrester-WMF) [07:08:44] 10Phabricator-Bot-Requests, 10Discourse, 10Space: Loop trying to create an account in Wikimedia Space with certain browser versions - https://phabricator.wikimedia.org/T226545 (10Whatamidoing-WMF) Saving dozens of tabs and restarting Safari seems to have solved the problem. Or perhaps it went away for unrel... [08:17:37] 10Phabricator: Change username from Baloch007 to Af420 - https://phabricator.wikimedia.org/T226676 (10Aklapper) [08:29:37] 10Project-Admins: Proposal for global tag: "Needs Discussion" - https://phabricator.wikimedia.org/T226671 (10Aklapper) No, please absolutely not. Ideally, project tags are for categorization information which is static over [most of] the lifespan of a task. Project tags are not to express temporary (non-stati... [08:58:35] (03CR) 10Hashar: [C: 03+2] "The debian-glue jobs only trigger when a change touch a file under the ./debian/ directory." [integration/config] - 10https://gerrit.wikimedia.org/r/519022 (https://phabricator.wikimedia.org/T226449) (owner: 10MarcoAurelio) [09:43:54] 10Project-Admins: Proposal for global tag: "Needs Discussion" - https://phabricator.wikimedia.org/T226671 (10Peachey88) Not weighing in on if a project is right or wrong. But a global project as proposed would be non descriptive for this purpose. If a task has multiple projects on it, Where does the needs discu... [09:44:57] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) I had a meeting with @thcipriani yesterday and we came up with the following conting... [09:47:20] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) [10:13:07] 10Continuous-Integration-Config: Setup CI for operations/debs/coredns.git - https://phabricator.wikimedia.org/T226700 (10MarcoAurelio) [12:04:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10jijiki) >>! In T224857#5288470, @Joe wrote: > - Whenver a rolling restart is needed, tha... [12:19:37] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10serviceops, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) >>! In T224857#5288802, @jijiki wrote: >>>! In T224857#5288470, @Joe wrote: > >> -... [13:57:45] (03PS1) 10Pablo Grass (WMDE): codehealth: send failures via email [integration/config] - 10https://gerrit.wikimedia.org/r/519403 [14:03:40] RECOVERY - Puppet staleness on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:04:53] RECOVERY - Free space - all mounts on deployment-eventlog05 is OK: OK: deployment-prep.deployment-eventlog05.diskspace._var_lib_mysql.byte_percentfree (No valid datapoints found) [14:05:04] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [14:05:36] RECOVERY - Puppet staleness on deployment-ms-fe03 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:06:54] RECOVERY - Free space - all mounts on deployment-hadoop-test-2 is OK: OK: All targets OK [14:06:57] RECOVERY - Puppet errors on deployment-kafka-main-2 is OK: OK: Less than 1.00% above the threshold [2.0] [14:07:12] RECOVERY - Puppet errors on deployment-wikifeeds01 is OK: OK: Less than 1.00% above the threshold [2.0] [14:07:41] RECOVERY - Free space - all mounts on deployment-etcd-01 is OK: OK: All targets OK [14:07:50] i created a foolish patch to address a challenge in working with codehealth jobs. any input welcome https://gerrit.wikimedia.org/r/c/integration/config/+/519403 [14:09:35] (03CR) 10Pablo Grass (WMDE): "This is to describe/address a challenge in working with codehealth jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/519403 (owner: 10Pablo Grass (WMDE)) [14:24:43] 10Phabricator-Bot-Requests, 10Discourse, 10Space (Jul-Sep-2019): Loop trying to create an account in Wikimedia Space with certain browser versions - https://phabricator.wikimedia.org/T226545 (10Qgil) [14:26:09] 10Project-Admins: Proposal for global tag: "Needs Discussion" - https://phabricator.wikimedia.org/T226671 (10MBinder_WMF) Good points, thanks, both. I will discuss with the team and hold off on a tag for the time being. :) [14:38:50] hmm, something is messed up [14:38:56] php7.2 is installed in the 7.3 container [14:39:04] (composer-package) [14:42:24] i think some of the php packages install php 7.1, 7.2 and 7.3 for compat. [14:42:53] there should be no need for php7.2-cli [14:43:38] hmm [14:43:45] seems like an old container version [14:43:46] weird [14:45:21] (03PS1) 10Legoktm: Bump composer-package-* docker images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/519417 [14:45:32] (03CR) 10Legoktm: [C: 03+2] Bump composer-package-* docker images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/519417 (owner: 10Legoktm) [14:47:53] (03Merged) 10jenkins-bot: Bump composer-package-* docker images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/519417 (owner: 10Legoktm) [15:06:27] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 3 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10MoritzMuehlenhoff) @mmodell I've built an i... [15:14:40] (03PS1) 10Legoktm: Install php-ast in composer-package-* containers [integration/config] - 10https://gerrit.wikimedia.org/r/519424 (https://phabricator.wikimedia.org/T218719) [15:15:27] (03CR) 10Legoktm: [C: 03+2] Install php-ast in composer-package-* containers [integration/config] - 10https://gerrit.wikimedia.org/r/519424 (https://phabricator.wikimedia.org/T218719) (owner: 10Legoktm) [15:16:18] (03CR) 10jerkins-bot: [V: 04-1] Install php-ast in composer-package-* containers [integration/config] - 10https://gerrit.wikimedia.org/r/519424 (https://phabricator.wikimedia.org/T218719) (owner: 10Legoktm) [15:16:22] oof [15:16:56] (03CR) 10jerkins-bot: [V: 04-1] Install php-ast in composer-package-* containers [integration/config] - 10https://gerrit.wikimedia.org/r/519424 (https://phabricator.wikimedia.org/T218719) (owner: 10Legoktm) [15:17:09] (03PS2) 10Legoktm: Install php-ast in composer-package-* containers [integration/config] - 10https://gerrit.wikimedia.org/r/519424 (https://phabricator.wikimedia.org/T218719) [15:19:24] (03CR) 10Legoktm: [C: 03+2] Install php-ast in composer-package-* containers [integration/config] - 10https://gerrit.wikimedia.org/r/519424 (https://phabricator.wikimedia.org/T218719) (owner: 10Legoktm) [15:21:30] (03Merged) 10jenkins-bot: Install php-ast in composer-package-* containers [integration/config] - 10https://gerrit.wikimedia.org/r/519424 (https://phabricator.wikimedia.org/T218719) (owner: 10Legoktm) [15:22:12] !log building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424 [15:22:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:25:57] [contint1001.wikimedia.org] out: 2019-06-27 15:24:50,456 [docker-pkg-build] INFO - /bin/sh: 1: make: not found [15:25:59] sighhhh [15:27:27] (03PS1) 10Legoktm: php-ast: Install build-essential too [integration/config] - 10https://gerrit.wikimedia.org/r/519428 [15:27:40] (03CR) 10Legoktm: [C: 03+2] php-ast: Install build-essential too [integration/config] - 10https://gerrit.wikimedia.org/r/519428 (owner: 10Legoktm) [15:29:12] (03Merged) 10jenkins-bot: php-ast: Install build-essential too [integration/config] - 10https://gerrit.wikimedia.org/r/519428 (owner: 10Legoktm) [15:29:47] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 - https://phabricator.wikimedia.org/T218719 (10Legoktm) >>! In T218719#5285204, @Daimona wrote: > At this point, we'd need php-ast 1.0.1 to be installed for the following jobs: > - composer-package-php70... [15:29:52] !log building composer-package-*, php-ast images for https://gerrit.wikimedia.org/r/519424 (try #2) [15:29:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:36:48] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 - https://phabricator.wikimedia.org/T218719 (10Daimona) >>! In T218719#5289370, @Legoktm wrote: > I assume leaving off php73 was accidental? Yes, thanks. > For composer-package-*, I'm going to go the ea... [15:36:51] (03PS1) 10Legoktm: php-ast: Use cp instead of COPY [integration/config] - 10https://gerrit.wikimedia.org/r/519430 [15:37:04] (03CR) 10Legoktm: [C: 03+2] php-ast: Use cp instead of COPY [integration/config] - 10https://gerrit.wikimedia.org/r/519430 (owner: 10Legoktm) [15:37:06] final try... [15:38:34] (03Merged) 10jenkins-bot: php-ast: Use cp instead of COPY [integration/config] - 10https://gerrit.wikimedia.org/r/519430 (owner: 10Legoktm) [15:43:59] (03PS1) 10Legoktm: docker: Bump composer-package-* image versions [integration/config] - 10https://gerrit.wikimedia.org/r/519433 [15:46:12] well that didn't work [15:47:47] (03CR) 10Legoktm: [C: 04-1] "11:45:35 PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/ast.so' - /usr/lib/php/20151012/ast.so: undefine" [integration/config] - 10https://gerrit.wikimedia.org/r/519433 (owner: 10Legoktm) [15:48:27] 10Continuous-Integration-Config, 10phan-taint-check-plugin, 10Patch-For-Review: Upgrade php-ast to 1.0.1 - https://phabricator.wikimedia.org/T218719 (10Legoktm) OK, there's some PHP version difference that I still need to debug... :| `11:45:35 PHP Warning: PHP Startup: Unable to load dynamic library '/usr/... [15:59:05] (03PS1) 10Hashar: Complete test for quibble.zuul.clone() [integration/quibble] - 10https://gerrit.wikimedia.org/r/519438 [15:59:46] (03CR) 10jerkins-bot: [V: 04-1] Complete test for quibble.zuul.clone() [integration/quibble] - 10https://gerrit.wikimedia.org/r/519438 (owner: 10Hashar) [16:05:00] (03CR) 10Hashar: "I found another solution using:" (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/519049 (owner: 10Awight) [16:07:35] (03CR) 10Jforrester: docker: Bump composer-package-* image versions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/519433 (owner: 10Legoktm) [16:12:55] James_F: T226724 ? can you help? [16:12:56] T226724: Gerrit admin permissions for Ottomata - https://phabricator.wikimedia.org/T226724 [16:20:35] (03PS2) 10Hashar: Complete test for quibble.zuul.clone() [integration/quibble] - 10https://gerrit.wikimedia.org/r/519438 [16:21:08] (03CR) 10Hashar: "Fixed flake8 issues" [integration/quibble] - 10https://gerrit.wikimedia.org/r/519438 (owner: 10Hashar) [16:21:56] ottomata: Sorry, no, that's an SRE thing. [16:24:52] opk [16:24:53] thanks [16:27:07] oh cool, it got sre-access-request tag added to it by herald, i see [16:27:17] Well. By me. :-) [16:27:20] oh! [16:27:35] Herald added "Operations". [16:27:38] oh oh [16:27:39] yes. [16:27:43] thank you! :) [16:27:49] Any time. :-) [16:41:25] James_F: Did you get anywhere with the webauthn tests not seemingly runnning the php tests unless you `check php`? [16:50:46] 10Continuous-Integration-Config: Add `ext-mysqli` to the CI container - https://phabricator.wikimedia.org/T226585 (10Jdforrester-WMF) Well, given you just added php-ast to those images, I guess this is fine. [16:51:46] James_F: ast was already in those images, I just switched the version (and failed at it), but I think adding mysqli is trivial and should be done [16:55:51] 10Continuous-Integration-Config, 10MediaWiki-extensions-OATHAuth, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Contractor - Doing): Enable Phan and seccheck on WebAuthn extension - https://phabricator.wikimedia.org/T226737 (10Reedy) [16:55:54] Reedy: Sorry, no, was distracted by other things. [16:56:03] legoktm: Sure. [16:56:19] legoktm: I just worry that the "composer" image is drifting far from its original purpose. [16:57:12] I think it is, but I'm not sure that's a bad thing. I used to be a fan of one container per purpose, but the tooling to manage so many containers is non-existent and provides very little benefit and takes a lot more work [16:58:39] like, I'm going to fold in the mediawiki-phan-seccheck image into the mediawiki-phan one because it's just an extra bash script [16:59:24] (03PS1) 10Reedy: Enable phan on WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/519454 (https://phabricator.wikimedia.org/T226737) [16:59:26] (03PS1) 10Reedy: Enable extension-seccheck-non-voting on WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/519455 (https://phabricator.wikimedia.org/T226737) [16:59:28] (03PS1) 10Reedy: Make seccheck voting on WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/519456 (https://phabricator.wikimedia.org/T226737) [17:01:30] (03CR) 10Reedy: [C: 04-2] "Probably needs other messing around as we don't want the PHP70 running on this extension" [integration/config] - 10https://gerrit.wikimedia.org/r/519456 (https://phabricator.wikimedia.org/T226737) (owner: 10Reedy) [17:02:12] (03CR) 10Reedy: [C: 03+2] Enable phan on WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/519454 (https://phabricator.wikimedia.org/T226737) (owner: 10Reedy) [17:03:37] (03Merged) 10jenkins-bot: Enable phan on WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/519454 (https://phabricator.wikimedia.org/T226737) (owner: 10Reedy) [17:04:10] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519454 [17:04:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:06:33] 10Continuous-Integration-Config: WebAuthn not running correct tests automatically - https://phabricator.wikimedia.org/T226738 (10Reedy) [17:07:25] oh, bleugh [17:07:32] phan is still on an older PHP too [17:09:01] Hang on... https://github.com/wikimedia/integration-config/blob/master/zuul/layout.yaml#L1132-L1138 [17:09:32] 18:05:58 - web-auth/webauthn-lib v1.2.0 requires php ^7.1 -> your PHP version (7.0.33) does not satisfy that requirement. [17:19:57] so uh [17:20:02] it's a little more complicated [17:20:15] because dependencies are installed using the ci-src-setup image iirc [17:33:24] lol [17:35:05] Wait, I moved phan to 72. [17:35:32] seccheck is still 70 though, yeah, that can't be voting yet. :-( [17:36:46] but that isn't seccheck I'm complaining about [17:36:56] I'd made the patch to use the non-voting one for the time being [17:37:07] Hmm. [17:37:22] https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/909/console [17:37:23] (03CR) 10Reedy: [C: 03+2] Enable extension-seccheck-non-voting on WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/519455 (https://phabricator.wikimedia.org/T226737) (owner: 10Reedy) [17:37:30] 18:05:55 + exec docker run --volume /srv/jenkins-workspace/workspace/mwext-php72-phan-docker [17:37:34] then bitches [17:39:48] (03Merged) 10jenkins-bot: Enable extension-seccheck-non-voting on WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/519455 (https://phabricator.wikimedia.org/T226737) (owner: 10Reedy) [17:39:48] Something's running docker-registry.wikimedia.org/releng/composer:0.1.9 which it should be using composer-php72. [17:39:48] Something for me to look at after the Activities meeting if you don't get there first. [17:39:48] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/519455 [17:39:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:40:25] name: 'php-composer-security-docker' [17:40:30] image: 'docker-registry.wikimedia.org/releng/composer:0.1.9' [17:41:15] Ah [17:41:15] name: docker-setup-mwext-for-phan [17:41:20] image: 'docker-registry.wikimedia.org/releng/composer:0.1.9' [17:41:36] Only 5 places using composer:0.1.9 [17:41:47] Oooh, I thought I upgraded that? [17:41:53] Specifically for that reason. [17:42:40] James_F: any idea what's up with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/513250? two +2's, but CI is quiet about merging it [17:43:47] kostajh: It depends on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/513268 which isn't merged. [17:47:34] (03PS1) 10Reedy: Use composer-php72:0.2.0 on docker-setup-mwext-for-phan [integration/config] - 10https://gerrit.wikimedia.org/r/519467 [17:47:55] Can that just be deployed via fab? [17:48:23] No, jjb macro -> via jjb. [17:50:54] James_F: oh right, of course. Thanks [17:52:14] kostajh: Gerrit goes out of its way to not make that clear, sadly. [18:06:44] (03CR) 10Reedy: [C: 03+2] Use composer-php72:0.2.0 on docker-setup-mwext-for-phan [integration/config] - 10https://gerrit.wikimedia.org/r/519467 (owner: 10Reedy) [18:07:50] no help there... [18:08:55] (03Merged) 10jenkins-bot: Use composer-php72:0.2.0 on docker-setup-mwext-for-phan [integration/config] - 10https://gerrit.wikimedia.org/r/519467 (owner: 10Reedy) [18:23:09] feels like there's something else that needs doing [18:24:04] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10thcipriani) Investigated this a bit today. I was hoping with 3 incidents in one day that the tri... [18:31:07] 10Continuous-Integration-Config: mwext-php72-phan-docker running some parts under PHP 7.0 - https://phabricator.wikimedia.org/T226752 (10Reedy) [18:31:26] 10Continuous-Integration-Config: mwext-php72-phan-docker running composer under PHP 7.0 - https://phabricator.wikimedia.org/T226752 (10Reedy) [18:40:35] 10Continuous-Integration-Config: mwext-php72-phan-docker running composer under PHP 7.0 - https://phabricator.wikimedia.org/T226752 (10Reedy) I thought https://gerrit.wikimedia.org/r/#/c/integration/config/+/519467/ would fix it... But still seems to be using the old version... ` $ ~/.local/bin/jenkins-jobs --... [18:51:56] Reedy: Let me have a look. [18:52:20] 10Continuous-Integration-Config, 10Release-Engineering-Team (Watching / External), 10Analytics, 10Patch-For-Review: Status of analytics/limn-*-data git repositories? - https://phabricator.wikimedia.org/T221064 (10Nuria) a:05Milimetric→03fdans [18:53:55] Reedy: Looks like the jjb jobs weren't updated? [18:54:03] I did [18:54:38] Or do dependant jobs also nee updating? [18:54:47] Reedy: php-composer-security-docker is still pointed at composer 0.1.9, did you mean to change that? [18:54:53] All the jobs need updating. [18:55:05] I've pushed all the phan ones. [18:55:10] I didn't touch that one, as I didn't think it was the offending one... [18:55:17] It probably isn't. [18:57:15] Reedy: New problem, it now wants ext-gmp [18:57:28] But the PHP version is now right. [18:57:35] That's definitely a different issue [18:57:59] Which is installed? [18:58:09] Synchronizing all jobs [18:58:09] This will easily take over an hour and might overwrite things you don't intend to. Do not do this. [18:58:12] mediawiki-phan has php7.2-gmp. [18:59:09] Reedy: yeah don't do that [18:59:43] 10Continuous-Integration-Config: mwext-php72-phan-docker running composer under PHP 7.0 - https://phabricator.wikimedia.org/T226752 (10Jdforrester-WMF) That was fixed (once I re-pushed the phan jjb jobs), but for some reason it now can't find the `gmp` extension? [19:10:38] James_F: Based on https://github.com/wikimedia/integration-config/search?q=php7.2-gmp&unscoped_q=php7.2-gmp [19:10:43] There's other packages missing too [19:10:45] gd for example [19:11:03] bcmath [19:12:47] * James_F re-reads the inheritance tree. [19:17:42] So we're executing in composer-php72:0.2.0 which is composer + php72; composer depends on php but (?) presumably php72 over-rides that. mwext-php72-phan-docker is pulled in as a volume but not run, right? [19:19:21] Beforehand, we were executing in composer:0.1.9 which depends on php which provides `php7.0-cli php7.0-zip php-ast php7.0-curl php7.0-dba php7.0-gd php7.0-mbstring php-redis php7.0-sqlite3 php-xdebug php7.0-xml` [19:20:13] php72 provides `php7.2-cli php7.2-zip php-ast php7.2-curl php7.2-dba php7.2-mbstring php-redis php7.2-sqlite3 php-xdebug php7.2-xml`, i.e. the same except for "php7.2-gd isn't installable, so it's skipped" [19:21:53] gmp is provided by mediawiki-phan, php56, and the quibble images, but not elsewhere. [19:22:01] So… how did this work beforehand? [19:22:47] Did it? [19:22:55] Or was nothing using the extensions [19:24:10] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T220736 (10jeena) [19:24:30] But this is just basic phan trying to use them, right? [19:24:37] Or is this new code that's newly trying to use them? [19:25:08] Ah, this repo's requirements are what is demanding this. [19:25:40] Anyway, sure, we can put ext-gmp everywhere, I think. [19:26:52] MW itself does call (conditionally) a lot of gmp stuff [19:27:16] I think we only installed them in php70 because of this reason tbh [19:34:33] OK, one mo. [19:34:50] <3 [19:35:12] legoktm: It's in quibble-stretch-php70 but not php70 (that is, php) unless I'm missing something? [19:36:00] I think we just forgot to add it everywhere else, is what I meant...let me find the phab ticket [19:37:38] https://phabricator.wikimedia.org/T185393 [19:38:46] Aha, so it's a regression [19:54:49] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+1] Some bug fixes [releng/local-charts] - 10https://gerrit.wikimedia.org/r/517780 (owner: 10Jeena Huneidi) [20:07:09] (03CR) 10Hashar: Codehealth: Skip MW install, generate code coverage from unit tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/518714 (owner: 10Kosta Harlan) [20:13:29] (03PS1) 10Jforrester: dockerfiles: Install php-gmp on all php base images, bump users [integration/config] - 10https://gerrit.wikimedia.org/r/519508 [20:13:41] legoktm: That's gd, not gmp… [20:14:07] (But gd is also missing from the php72 images, apparently? Is that an issue?) [20:15:11] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: Install php-gmp on all php base images, bump users [integration/config] - 10https://gerrit.wikimedia.org/r/519508 (owner: 10Jforrester) [20:17:33] (03PS2) 10Jforrester: dockerfiles: Install php-gmp on all php base images, bump users [integration/config] - 10https://gerrit.wikimedia.org/r/519508 [20:41:05] James_F: clearly I'm not awake today [20:41:29] legoktm: :-) How scary does https://gerrit.wikimedia.org/r/519508 look to you? I'm a bit wary… [20:44:25] big patch [20:44:36] coffee for Kunal :D [20:45:51] James_F: I suspect you're also going to bring in PHP patch releases by rebuilding all the images, which should be a good thing, but something to be aware of it stuff randomly starts failing [20:46:19] legoktm: Hmm, fair point. But oh well. [20:46:37] (Do we want to push any other items into the general images whilst we're at it?) [20:47:02] I was going to suggest that, do you also want to add bcmath, gd and whatever else Reedy mentioned earlier? [20:47:25] gmp [20:47:30] oh [20:47:35] :-D [20:47:37] Yes dear. [20:47:38] compare https://github.com/wikimedia/integration-config/search?q=php7.2-gmp&unscoped_q=php7.2-gmp and... [20:47:53] ~13 to double check [20:48:19] Obviously I can upstream all the ones in quibble into php main, but… [20:48:31] Presumably there was a reason we didn't? [20:51:06] My guess is that nothing explicitly needed it [20:51:15] Hmm. [20:51:28] Should we push php-mysql into them all whilst we're at it? [20:52:07] Also, is Debian clever enough to install the right version of the PHP extension to go with the version of PHP? [20:54:52] yes kind of [20:55:08] the best response [20:55:43] if it's a core PHP extension, then you should install php7.0-whatever (check the sidebar of https://tracker.debian.org/pkg/php7.0) otherwise there will just be php-foo packages, which should have the right versions available [20:56:00] Because e.g. composer-php71 composites php71 (with the right version of gmp, presumably?) and composer (which itself is based on php i.e. php70 and so has the "wrong" version of gmp). [20:56:16] Which one "wins"? [20:56:37] well it depends how it uses composer [20:56:44] I think it's just copying stuff right? [20:57:26] Yeah, just copying the composer binary, it looks like. [20:57:28] yeah, just copying stuff [20:58:22] Eh, screw it. I'll be selective in pushing the jjb entries to latest. Worse comes to the worst, we can just skip these versions entirely. [20:58:26] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Install php-gmp on all php base images, bump users [integration/config] - 10https://gerrit.wikimedia.org/r/519508 (owner: 10Jforrester) [20:59:59] (03Merged) 10jenkins-bot: dockerfiles: Install php-gmp on all php base images, bump users [integration/config] - 10https://gerrit.wikimedia.org/r/519508 (owner: 10Jforrester) [21:00:24] * legoktm afk [21:09:31] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 3 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10mmodell) Thanks @MoritzMuehlenhoff ! I rea... [21:27:25] !log Pushing new docker php base images with gmp, and all their descendants [21:27:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:49:52] (03CR) 10Hashar: "I am pretty sure this change cause the gmp extension to no more be part of the quibble containers :-\" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/519508 (owner: 10Jforrester) [21:50:39] (03CR) 10Jforrester: "> Patch Set 2:" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/519508 (owner: 10Jforrester) [21:51:33] Fun, "ERROR: image docker-registry.discovery.wmnet/releng/composer-php56 failed to build, see logs for details". [21:51:52] Not that we want to use php56 anywhere anyway.:-) [21:52:02] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T220736 (10Krinkle) [22:03:56] (03CR) 10Hashar: "My only concern now is that the Google apt repository get us the latest stable version of Bazel currently available. But we might not nee" [integration/config] - 10https://gerrit.wikimedia.org/r/493638 (owner: 10Paladox) [23:04:43] Oh, bother. [23:04:53] ! [remote rejected] HEAD -> refs/for/master (the number of pushed changes in a batch exceeds the max limit 20) [23:05:59] (03PS1) 10Jforrester: jjb: [integration-composer-check-php72-docker] Bump to php image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519548 [23:06:01] (03PS1) 10Jforrester: jjb: [mwext-php72-phan-docker] Bump to mediawiki-phan image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519549 [23:06:03] (03PS1) 10Jforrester: jjb: [mediawiki-core-php72-phan-docker] Bump to mediawiki-phan image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519550 [23:06:05] (03PS1) 10Jforrester: jjb: [php72-phan-docker] Bump to mediawiki-phan image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519551 [23:06:07] (03PS1) 10Jforrester: jjb: [mwext-php72-phan-seccheck-docker] Bump to mediawiki-phan-seccheck image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519552 [23:06:09] (03PS1) 10Jforrester: jjb: [mediawiki-core-php72-phan-seccheck-docker] Bump to mediawiki-phan-seccheck image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519553 [23:06:11] (03PS1) 10Jforrester: jjb: [php-composer-security-docker] Bump to composer image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519554 [23:06:13] (03PS1) 10Jforrester: jjb: [composer-package-7\d-docker] Bump to composer-package images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519555 [23:06:15] (03PS1) 10Jforrester: jjb: [composer-7\d-docker] Bump to composer-test images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519556 [23:06:17] (03PS1) 10Jforrester: jjb: [mwgate-composer-7\d-docker] Bump to composer-test images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519557 [23:06:19] (03PS1) 10Jforrester: jjb: [mediawiki-core-phpmetrics-docker] Bump to phpmetrics image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519558 [23:06:21] (03PS1) 10Jforrester: jjb: [phpunit-coverage-docker-publish] Bump to composer-php72 image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519559 [23:06:23] (03PS1) 10Jforrester: jjb: [oojs-ui-docker-publish] Bump to npm-test-oojsuiimage with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519560 [23:06:25] (03PS1) 10Jforrester: jjb: [oojs-ui-npm-run-jenkins-node-6-docker] Bump to npm-test-oojsui image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519561 [23:06:27] (03PS1) 10Jforrester: jjb: [wikimedia-cz-mediawiki-config] Bump to composer-test image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519562 [23:06:29] (03PS1) 10Jforrester: jjb: [wikiba.se-composer-php72-docker] Bump to composer-test-php72 image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519563 [23:06:31] (03PS1) 10Jforrester: jjb: [operations-mw-config-php7\d-composer-lint-docker] Bump to composer images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519564 [23:06:35] There we go. [23:06:41] (03PS1) 10Jforrester: jjb: [parsoidsvc-composer-package-php7\d-docker] Bump to composer-package images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519566 [23:06:45] (03PS1) 10Jforrester: jjb: [wikimedia-fundraising-crm-composer-php72-docker] Bump to composer-test-php72 image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519567 [23:06:47] (03PS1) 10Jforrester: jjb: [wikimedia-fundraising-civicrm-docker] Bump to civicrm image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519568 [23:06:49] (03PS1) 10Jforrester: jjb: [mw-tools-codesniffer-mwcore-testrun] Bump to composer image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519569 [23:06:51] (03PS1) 10Jforrester: jjb: [docker-ci-src-setup-mw] Bump to ci-src-setup image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519570 [23:10:11] James_F :D [23:12:25] Well, l.egoktm *did* say to deploy it carefully. :-) [23:15:34] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 64.29% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:22:20] (03PS1) 10Jforrester: dockerfiles: Re-install php-gmp on all quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/519572 [23:22:22] (03PS1) 10Jforrester: jjb: Point quibble jobs at latest images (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/519573 [23:22:53] (03CR) 10Jforrester: "Quibble bits reverted in I81d9039626dff16248feb267b951963703e5c09a (built but never used)." [integration/config] - 10https://gerrit.wikimedia.org/r/519508 (owner: 10Jforrester) [23:25:16] (03CR) 10Jforrester: "recheck" [integration/composer] - 10https://gerrit.wikimedia.org/r/436960 (owner: 10Legoktm) [23:26:13] (03CR) 10Jforrester: [C: 03+2] "Deployed, works." [integration/config] - 10https://gerrit.wikimedia.org/r/519548 (owner: 10Jforrester) [23:28:33] (03Merged) 10jenkins-bot: jjb: [integration-composer-check-php72-docker] Bump to php image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519548 (owner: 10Jforrester) [23:28:54] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Mail, 10Operations, and 2 others: Phabricator email comments not posted - https://phabricator.wikimedia.org/T224752 (10Dzahn) nice fix! sorry, i was off for quite some time and reading this now. indeed a good catch that we did not catch during the migra... [23:31:00] (03CR) 10Jforrester: [C: 03+2] "Deployed, works." [integration/config] - 10https://gerrit.wikimedia.org/r/519549 (owner: 10Jforrester) [23:33:17] (03Merged) 10jenkins-bot: jjb: [mwext-php72-phan-docker] Bump to mediawiki-phan image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519549 (owner: 10Jforrester) [23:33:33] (03CR) 10Jforrester: [C: 03+2] jjb: [mediawiki-core-php72-phan-docker] Bump to mediawiki-phan image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519550 (owner: 10Jforrester) [23:35:46] (03Merged) 10jenkins-bot: jjb: [mediawiki-core-php72-phan-docker] Bump to mediawiki-phan image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519550 (owner: 10Jforrester) [23:37:45] (03CR) 10Jforrester: [C: 03+2] "We should probably scrap this job (no users except Parsoid where they have a custom job instead), but for now…" [integration/config] - 10https://gerrit.wikimedia.org/r/519551 (owner: 10Jforrester) [23:39:57] (03Merged) 10jenkins-bot: jjb: [php72-phan-docker] Bump to mediawiki-phan image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519551 (owner: 10Jforrester) [23:41:41] (03CR) 10Jforrester: [C: 03+2] "Fails in the "right" place." [integration/config] - 10https://gerrit.wikimedia.org/r/519552 (owner: 10Jforrester) [23:41:48] (03CR) 10Jforrester: [C: 03+2] jjb: [mediawiki-core-php72-phan-seccheck-docker] Bump to mediawiki-phan-seccheck image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519553 (owner: 10Jforrester) [23:43:59] (03Merged) 10jenkins-bot: jjb: [mwext-php72-phan-seccheck-docker] Bump to mediawiki-phan-seccheck image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519552 (owner: 10Jforrester) [23:44:06] (03CR) 10Jforrester: [C: 03+2] jjb: [php-composer-security-docker] Bump to composer image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519554 (owner: 10Jforrester) [23:44:08] (03Merged) 10jenkins-bot: jjb: [mediawiki-core-php72-phan-seccheck-docker] Bump to mediawiki-phan-seccheck image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519553 (owner: 10Jforrester) [23:46:18] legoktm: Yeah, looks like the silent upgrade broke composer-package-php70-docker for php-ast, at least. :-( [23:46:20] (03Merged) 10jenkins-bot: jjb: [php-composer-security-docker] Bump to composer image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519554 (owner: 10Jforrester) [23:46:50] 16:45:18 PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/ast.so' - /usr/lib/php/20151012/ast.so: undefined symbol: zend_string_init_interned in Unknown on line 0 [23:47:12] Presumably from f8a4334d49683fd6ef433779853d9f51a5f02c92 ? [23:48:22] (03PS1) 10Jforrester: layout: [parsoid] Drop php72-phan-docker experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/519576 [23:48:24] (03PS1) 10Jforrester: jjb: Drop php72-phan-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/519577 [23:51:32] (03PS2) 10Jforrester: dockerfiles: Re-install php-gmp on all quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/519572 [23:51:34] (03PS2) 10Jforrester: jjb: Point quibble jobs at latest images (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/519573 [23:51:36] (03PS2) 10Jforrester: layout: [parsoid] Drop php72-phan-docker experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/519576 [23:51:38] (03PS2) 10Jforrester: jjb: Drop php72-phan-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/519577 [23:51:40] (03PS2) 10Jforrester: jjb: [mwgate-composer-7\d-docker] Bump to composer-test images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519557 [23:51:42] (03PS2) 10Jforrester: jjb: [mediawiki-core-phpmetrics-docker] Bump to phpmetrics image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519558 [23:51:44] (03PS2) 10Jforrester: jjb: [phpunit-coverage-docker-publish] Bump to composer-php72 image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519559 [23:51:46] (03PS2) 10Jforrester: jjb: [oojs-ui-docker-publish] Bump to npm-test-oojsuiimage with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519560 [23:51:48] (03PS2) 10Jforrester: jjb: [oojs-ui-npm-run-jenkins-node-6-docker] Bump to npm-test-oojsui image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519561 [23:51:50] (03PS2) 10Jforrester: jjb: [wikimedia-cz-mediawiki-config] Bump to composer-test image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519562 [23:51:52] (03PS2) 10Jforrester: jjb: [wikiba.se-composer-php72-docker] Bump to composer-test-php72 image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519563 [23:51:54] (03PS2) 10Jforrester: jjb: [operations-mw-config-php7\d-composer-lint-docker] Bump to composer images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519564 [23:51:56] (03PS2) 10Jforrester: jjb: [parsoidsvc-composer-package-php7\d-docker] Bump to composer-package images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519566 [23:51:58] (03PS2) 10Jforrester: jjb: [wikimedia-fundraising-crm-composer-php72-docker] Bump to composer-test-php72 image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519567 [23:52:00] (03PS2) 10Jforrester: jjb: [wikimedia-fundraising-civicrm-docker] Bump to civicrm image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519568 [23:52:02] (03PS2) 10Jforrester: jjb: [mw-tools-codesniffer-mwcore-testrun] Bump to composer image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519569 [23:52:04] (03PS2) 10Jforrester: jjb: [docker-ci-src-setup-mw] Bump to ci-src-setup image with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519570 [23:52:06] (03PS2) 10Jforrester: jjb: [composer-package-php7\d-docker] Bump to composer-package images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519555 [23:52:08] (03PS2) 10Jforrester: jjb: [composer-php7\d-docker] Bump to composer-test images with gmp [integration/config] - 10https://gerrit.wikimedia.org/r/519556 [23:52:10] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Re-install php-gmp on all quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/519572 (owner: 10Jforrester) [23:52:13] James_F lol you had that many patches locally? [23:52:16] (03CR) 10Jforrester: [C: 04-1] "Broken due to php-ast installation that hadn't been rolled out yet." [integration/config] - 10https://gerrit.wikimedia.org/r/519555 (owner: 10Jforrester) [23:52:21] (03CR) 10Jforrester: [C: 04-1] "Broken due to php-ast installation that hadn't been rolled out yet." [integration/config] - 10https://gerrit.wikimedia.org/r/519556 (owner: 10Jforrester) [23:52:23] that must have been a pain to change :P [23:52:56] paladox: `git rebase -i` is your friend. [23:53:01] oh [23:53:14] Only 29 patches in my local stack. [23:53:18] ;-) [23:53:55] lol [23:59:06] (03Merged) 10jenkins-bot: dockerfiles: Re-install php-gmp on all quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/519572 (owner: 10Jforrester)