[00:00:31] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:00:42] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:01:12] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:03:07] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:05:09] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:05:35] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:06:44] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:07:10] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:08:05] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:08:15] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:12:35] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:14:17] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:15:53] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:16:06] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:16:15] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:17:15] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:18:19] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:19:24] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:19:49] uh [00:19:53] wtf is that [00:21:11] ugh [00:21:17] more scap package stuff [00:23:10] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:47:35] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:48:01] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:48:13] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [00:54:17] RECOVERY - Puppet errors on deployment-cassandra3-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:54:39] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:54:55] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:11] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:31] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:35] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:41] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:55:55] RECOVERY - Puppet errors on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [0.0] [00:56:07] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [00:56:12] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:56:15] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:56:43] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:57:09] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [00:57:15] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [00:57:21] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:07] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:09] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:19] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:59:26] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [01:14:10] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [01:19:25] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:45:24] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:45:32] !log deployed to beta: [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba [04:45:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:08:58] wikibugs is dead? [09:47:55] 10Release-Engineering-Team (Kanban), 10Graphs, 10VisualEditor, 10Patch-For-Review: [Graph] extension fails QUnit tests due to timeout - https://phabricator.wikimedia.org/T198229#4318526 (10hashar) a:03hashar [09:51:52] 10Release-Engineering-Team (Kanban), 10Graphs, 10VisualEditor, 10Patch-For-Review: [Graph] extension fails QUnit tests due to timeout - https://phabricator.wikimedia.org/T198229#4318530 (10hashar) [10:11:51] (03PS1) 10Prtksxna: Add job for design landing page [integration/config] - 10https://gerrit.wikimedia.org/r/442266 [10:12:55] (03CR) 10Prtksxna: "Just need to run `npm test` to run stylelint." [integration/config] - 10https://gerrit.wikimedia.org/r/442266 (owner: 10Prtksxna) [10:47:19] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<22.22%) [10:49:36] (03CR) 10Hashar: [C: 032] Add job for design landing page [integration/config] - 10https://gerrit.wikimedia.org/r/442266 (owner: 10Prtksxna) [10:50:42] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-2.diskspace.root.byte_percentfree (<20.00%) [10:51:16] (03Merged) 10jenkins-bot: Add job for design landing page [integration/config] - 10https://gerrit.wikimedia.org/r/442266 (owner: 10Prtksxna) [12:06:28] 10Phabricator, 10Research: Make new tasks within a specific project use a template in description field - https://phabricator.wikimedia.org/T91538#4319094 (10Aklapper) 05stalled>03declined Unfortunately closing this report as declined as we have not seen input from #Research or @ggellerman. If this is stil... [12:14:10] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [12:16:55] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [12:18:13] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:20:26] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:23:20] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:24:11] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:26:42] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:27:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:30:38] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:30:55] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:32:11] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:34:07] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:36:10] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:37:34] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:37:46] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:38:10] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:38:36] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:39:00] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:39:11] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:41:00] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4319158 (10hashar) [12:44:17] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:45:18] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:47:06] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:47:14] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:57:34] (03PS1) 10Hashar: GoogleAppEngine requires composer [integration/config] - 10https://gerrit.wikimedia.org/r/442294 (https://phabricator.wikimedia.org/T196346) [12:58:17] (03CR) 10Hashar: [C: 032] GoogleAppEngine requires composer [integration/config] - 10https://gerrit.wikimedia.org/r/442294 (https://phabricator.wikimedia.org/T196346) (owner: 10Hashar) [12:59:34] (03Merged) 10jenkins-bot: GoogleAppEngine requires composer [integration/config] - 10https://gerrit.wikimedia.org/r/442294 (https://phabricator.wikimedia.org/T196346) (owner: 10Hashar) [13:54:49] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4319350 (10hashar) [14:01:38] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319373 (10zeljkofilipin) [14:03:22] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4313098 (10zeljkofilipin) [14:04:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4319393 (10hashar) [14:09:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319404 (10zeljkofilipin) [14:10:17] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4319417 (10hashar) [14:10:42] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4313098 (10zeljkofilipin) @hashar all... [14:10:49] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4278980 (10hashar) [14:14:35] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319430 (10zeljkofilipin) [14:16:32] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4313098 (10zeljkofilipin) >>! In T198... [14:22:15] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319443 (10zeljkofilipin) [[ https://... [14:28:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319447 (10zeljkofilipin) There isn't... [14:30:11] James_F: this is kind of a shot in the dark but... Do you happen to know how to get into the @wikimedia NPM organization? We have a couple repos we're transitioning from personal hosting to Wikimedia (T197251) and I'd like to publish their artifacts under @wikimedia/ ideally. [14:30:11] T197251: Transition mw-node-qunit and resource-modules to wikimedia/ - https://phabricator.wikimedia.org/T197251 [14:39:07] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319477 (10zeljkofilipin) [[ URL | 44... [14:41:27] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319489 (10zeljkofilipin) [14:48:11] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqiad: tin has a failing hdd - https://phabricator.wikimedia.org/T174449#4319499 (10Cmjohnson) 05stalled>03Resolved a:03Cmjohnson This server now has a decom task https://phabricator.wikimedia.org/T196175 [14:53:47] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319512 (10WMDE-leszek) Well, that pa... [15:13:07] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319570 (10zeljkofilipin) >>! In T198... [15:18:50] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319580 (10WMDE-leszek) Regarding #3... [15:29:08] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319606 (10WMDE-leszek) > #3 quibble-... [15:31:10] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319624 (10WMDE-leszek) Regarding scr... [15:31:22] https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/5918/console - build timeout. Known or already reported? [15:37:51] (03CR) 10Alexandros Kosiaris: [C: 04-1] "LGTM, apart from 1 comment" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/425936 (https://phabricator.wikimedia.org/T188935) (owner: 10Dduvall) [15:38:23] 10Scap (Scap3-MediaWiki-MVP), 10Operations, 10Wikimedia-Incident: Scap sync --restart not working - https://phabricator.wikimedia.org/T198185#4319638 (10fgiunchedi) [15:38:29] 10Scap, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: Update Debian Package for Scap3 to 3.8.3-1 - https://phabricator.wikimedia.org/T198277#4319635 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi Done! [15:40:31] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Wikimedia-Incident: Scap sync --restart not working - https://phabricator.wikimedia.org/T198185#4319649 (10thcipriani) 05Open>03Resolved a:03thcipriani Removed in Scap 3.8.3-1 which was just made live in production. [15:55:10] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10User-zeljkofilipin, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4319682 (10Pablo-WMDE) > #3 quibble-v... [16:00:23] niedzielski: I assume you meant GitHub @wikimedia, not NPM @wikimedia. [16:00:56] niedzielski: To do that, you first need to transfer it from your personal account to one of the org owners (e.g. James or myself), and then from there it can be moved again to the org. [16:01:16] GitHub will retain redirects all the way for everything, so no worries there :) [16:02:30] Krinkle: no the NPM @wikimedia organization. We want some packages to be installed from @wikimedia/foo @wikimedia/bar and so forth using NPM orgs (e.g., https://www.npmjs.com/docs/orgs/). [16:02:48] Ah you mean scoped packages instead of regular package names. [16:03:44] I assume you'd want to keep the same package name as now (e.g. no user/org prefix) [16:05:21] Hm.. https://www.npmjs.com/org/wikimedia [16:05:37] I see packages there that don't have scoped names like https://www.npmjs.com/package/grunt-banana-checker [16:06:24] I don't know what makes them appear there. There's nothing we do for that in package.json or publication process. It's also not an access control thing (there is no ~wikimedia user, and if there was, it doens't have rights over that package) [16:06:37] Possibly based on the github url for discovery purposes only? [16:06:52] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-extensions-LdapAuthentication, 10MediaWiki-extensions-OpenStackManager: OpenStackManager tests fail on Quibble container: Error: Call to undefined function ldap_bind() - https://phabricator.wikimedia.org/T198336#4319711 (10hashar) [16:07:44] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4319721 (10hashar) [16:09:37] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:23:35] (03PS1) 10Hashar: docker: add bcmath/ldap to Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/442338 (https://phabricator.wikimedia.org/T196346) [16:24:22] (03CR) 10Hashar: [C: 032] docker: add bcmath/ldap to Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/442338 (https://phabricator.wikimedia.org/T196346) (owner: 10Hashar) [16:25:40] (03Merged) 10jenkins-bot: docker: add bcmath/ldap to Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/442338 (https://phabricator.wikimedia.org/T196346) (owner: 10Hashar) [16:25:55] (03PS1) 10Hashar: Bump php7/hhvm Quibble jobs to 0.0.19-1 [integration/config] - 10https://gerrit.wikimedia.org/r/442339 (https://phabricator.wikimedia.org/T196346) [16:27:14] !log Building Docker containers releng/quibble-jessie-php55:0.0.19-1 and releng/quibble-stretch:0.0.19-1 | T196346 T198336 [16:27:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:27:18] T196346: [GoogleAppEngine] exception when saving page - https://phabricator.wikimedia.org/T196346 [16:27:18] T198336: OpenStackManager tests fail on Quibble container: Error: Call to undefined function ldap_bind() - https://phabricator.wikimedia.org/T198336 [16:27:38] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-LdapAuthentication, 10MediaWiki-extensions-OpenStackManager, 10Patch-For-Review: OpenStackManager tests fail on Quibble container: Error: Call ... - https://phabricator.wikimedia.org/T198336#4319762 [16:27:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-GoogleAppEngine, 10Patch-For-Review: [GoogleAppEngine] exception when saving page - https://phabricator.wikimedia.org/T196346#4319764 (10hashar) a:03hashar [16:33:33] hasharAway: It seems some of the builds are failing with Memory allocation errors. [16:33:44] Can you check if something is up with the underlying infra, or should it all be good? [16:33:51] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/442326/ [16:37:03] niedzielski: Aha, sure. [17:04:01] James_F Krinkle: Sorry-- was in a meeting. I need membership to the wikimedia org to publish packages under @wikimedia/ as I understand it. [17:04:20] We'd like the initial packages to be like @wikimedia/mw-node-qunit. [17:04:35] niedzielski: See my comment on the task. :-) [17:05:02] niedzielski: I can add you too. [17:05:03] James_F: thanks! [17:05:36] James_F: ah! That would be perfect. I'm "niedzielski" on NPM too. [17:06:51] niedzielski: Done. [17:08:11] \o/ [17:08:31] Thanks James_F Krinkle! [17:11:02] zeljkof: Hey, regarding https://phabricator.wikimedia.org/T198201 Is the LocalSettings.php (in selenium folder) being loaded in docker tests? [17:11:42] It seems not, otherwise it would hit https://ores.wikimedia.org/v3/scores/teswiki and not [https://ores.wikimedia.org/v3/scores/wikidb [17:21:48] the quibble-vendor job is timing out after 30min 3 times in a row at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/442326/ [17:21:51] Not sure what's going on there. [17:23:46] Krinkle: It always times out at 30 mins if it takes that long. Has someone added some new tests (e.g. a big new MCR test suite just landing)? [17:24:27] nah, the composer based one is passing and running the same subset of phpunit tests [17:24:38] Hmm. [17:24:57] They have in common that they use php70. [17:25:03] (the failing jobs) [17:25:41] the php70 spend 27min doing phpunit tests [17:26:13] the hhvm one, spends 5+8min [17:26:53] Yeah, strange. [17:27:24] the hhvm quibble job spends 5min in phpunit--without-db, and 8min in phpunit--db-only. the php70 job spends 27min in phpunit--without-dband subsequently times out [17:28:20] Ah, I got it wrong. the php70 job first halts for 23min doing nothing in the middle of composer-install and npm-install [17:28:25] looks like some kind of VM freeze [17:28:45] could be related to that memory issue earlier, but I don't know. someone who knows the infra more should look at that [17:30:03] (03CR) 10Hashar: "I have not refreshed the Jenkins jobs yet." [integration/config] - 10https://gerrit.wikimedia.org/r/442339 (https://phabricator.wikimedia.org/T196346) (owner: 10Hashar) [17:31:18] Krinkle: something is off in the infrastructure somewhere. The npm install spurts " ERR! registry error parsing json" and takes like 20 minutes [17:36:46] one would have to ding in the cloud infra and see what is going on there [17:39:26] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:57:23] Amir1: on the phone, will take a look tomorrow [17:59:13] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [17:59:27] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:18:41] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10ChangeProp, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#4320007 (10Mholloway) [18:34:49] hasharAway: Any news? It's affecting mw/core@master gate pipeline as well, not just wmf.10 [18:34:58] Essentially unable to test or merge in core. [18:37:27] hasharAway: https://github.com/npm/npm/wiki/Troubleshooting#invalid-json [18:37:38] Looks like it could be an upstream issue, but it might also be a castor/cache issue. [18:37:46] Is there a way to purge that? [18:41:32] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4320058 (10kaldari) Hey y'all, it's been 2 weeks and I still can't use my gerrit account. I've set up a temporary account, but I thought "temporary" meant a day, not 2 weeks. I... [18:42:34] kaldari: last time I checked your gerrit account had no email attached to it [18:57:16] Krinkle: I was with kids and greg for the weekly checkin so nothing done on my side [19:00:34] I cant even access grafana-admin anymore [19:00:55] hasharAway: use the regular grafana instead. ops deprecated -admin yesterday. native LDAP now. [19:01:44] doh [19:01:52] not sure how you manage to catch all those informations (it works) [19:12:49] 10Continuous-Integration-Infrastructure: MediaWiki build fails with "Build timed out (after 30 minutes)" - https://phabricator.wikimedia.org/T198346#4320087 (10Tgr) [19:16:40] !sal [19:16:40] https://tools.wmflabs.org/sal/releng [19:17:50] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4320114 (10thcipriani) >>! In T197083#4320058, @kaldari wrote: > Hey y'all, it's been 2 weeks and I still can't use my gerrit account. I've set up a temporary account, but I th... [19:18:27] 10Continuous-Integration-Infrastructure, 10Cloud-VPS: CI jobs takes too long / instances overloaded - https://phabricator.wikimedia.org/T198348#4320116 (10hashar) [19:18:32] Krinkle: I have filled https://phabricator.wikimedia.org/T198348 to track the issue [19:23:01] 10Continuous-Integration-Infrastructure, 10Cloud-VPS: CI jobs takes too long / instances overloaded - https://phabricator.wikimedia.org/T198348#4320132 (10hashar) [19:28:58] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320155 (10dduvall) [19:43:58] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T191056#4320232 (10dduvall) Rolling back group1 due to {T198350} blocker. [19:48:35] 10Continuous-Integration-Infrastructure, 10Cloud-VPS: CI jobs takes too long / instances overloaded - https://phabricator.wikimedia.org/T198348#4320241 (10hashar) An example of `npm install`, time is elapsed time in hh:mm:ss since start of build: ``` 00:06:28.594 INFO:test.run_extskin:Running "npm test" for Ci... [19:49:04] 10Continuous-Integration-Infrastructure, 10Wikimedia-log-errors (Shared Build Failure): MediaWiki build fails with "Build timed out (after 30 minutes)" - https://phabricator.wikimedia.org/T198346#4320245 (10Jdforrester-WMF) [19:49:33] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10Wikimedia-log-errors (Shared Build Failure): CI jobs takes too long / instances overloaded - https://phabricator.wikimedia.org/T198348#4320246 (10Krinkle) [19:49:37] 10Continuous-Integration-Infrastructure, 10Wikimedia-log-errors (Shared Build Failure): MediaWiki build fails with "Build timed out (after 30 minutes)" - https://phabricator.wikimedia.org/T198346#4320251 (10Krinkle) [19:49:40] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10Wikimedia-log-errors (Shared Build Failure): CI jobs takes too long / instances overloaded - https://phabricator.wikimedia.org/T198348#4320116 (10Krinkle) [19:50:38] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10Wikimedia-log-errors (Shared Build Failure): CI jobs takes too long / instances overloaded - https://phabricator.wikimedia.org/T198348#4320253 (10Jdforrester-WMF) p:05Triage>03High Not sure if this is quite at UBN yet, but it's at least High. [20:06:57] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10Wikimedia-log-errors (Shared Build Failure): CI jobs takes too long / instances overloaded - https://phabricator.wikimedia.org/T198348#4320367 (10hashar) The MediaWiki jobs are running integration-slave-docker-1001 to 1015 which is the [[ https://integr... [20:18:17] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4320431 (10Ladsgroup) Seco... [20:35:21] oh https://gerrit-review.googlesource.com/Documentation/pg-plugin-endpoints.html#_settings_screen yay [20:35:36] looks like we will be able to define more themes for polygerrit! [20:36:13] 10Release-Engineering-Team (Kanban), 10ORES, 10Quibble, 10Browser-Tests, and 4 others: ORES webdriver.io selenium test fail on CI due to lack of ORES server - https://phabricator.wikimedia.org/T198201#4320471 (10hashar) a:03hashar That is a Quibble issue. Thank you @Ladsgroup ! [20:36:34] 10Release-Engineering-Team (Kanban), 10ORES, 10Quibble, 10Browser-Tests, and 4 others: Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server) - https://phabricator.wikimedia.org/T198201#4320478 (10hashar) [21:13:10] oh about time that google issue tracker has a new ui! [21:15:58] oh nvm [21:16:06] just one of the features has [21:25:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T191056#4320568 (10Jdforrester-WMF) [21:27:13] Project mwext-phpunit-coverage-publish build #5943: 15ABORTED in 29 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5943/ [21:27:19] Project mwext-phpunit-coverage-publish build #5944: 15ABORTED in 6.3 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5944/ [21:27:33] Project mwext-phpunit-coverage-publish build #5945: 15ABORTED in 13 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5945/ [21:27:46] Project mwext-phpunit-coverage-publish build #5946: 15ABORTED in 12 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5946/ [21:28:11] Project mwext-phpunit-coverage-publish build #5947: 15ABORTED in 25 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5947/ [21:28:19] Project mwext-phpunit-coverage-publish build #5948: 15ABORTED in 7.1 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5948/ [21:28:31] Project mwext-phpunit-coverage-publish build #5949: 15ABORTED in 12 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5949/ [21:28:45] Project mwext-phpunit-coverage-publish build #5950: 15ABORTED in 13 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5950/ [21:45:03] Project mwext-phpunit-coverage-publish build #5960: 15ABORTED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5960/ [21:45:12] Project mwext-phpunit-coverage-publish build #5961: 15ABORTED in 8.5 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5961/ [21:45:40] Project mwext-phpunit-coverage-publish build #5962: 15ABORTED in 27 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/5962/ [21:50:13] (^ the above is me clearing out post-merge builds from l10nbot commits) [21:51:29] Thanks, Krinkle.