[02:30:32] Project beta-scap-eqiad build #222871: 04FAILURE in 0.84 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222871/ [02:47:59] Yippee, build fixed! [02:47:59] Project beta-scap-eqiad build #222872: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222872/ [03:45:02] !log Update cxserver to d913793 [03:45:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:18:32] 10Continuous-Integration-Infrastructure: mediawiki-phpunit-coverage-patch-docker missing HTML report artifact - https://phabricator.wikimedia.org/T205235 (10Legoktm) [05:34:57] Project beta-scap-eqiad build #222883: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222883/ [05:35:17] 05:34:57 05:34:57 Last output: [05:35:17] 05:34:57 sudo: a password is required [05:35:54] Project beta-scap-eqiad build #222884: 04STILL FAILING in 4.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222884/ [05:47:35] (03PS1) 10Legoktm: Add experimental mediawiki-core-php70-phan-seccheck-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/462341 (https://phabricator.wikimedia.org/T203630) [05:48:52] (03CR) 10jerkins-bot: [V: 04-1] Add experimental mediawiki-core-php70-phan-seccheck-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/462341 (https://phabricator.wikimedia.org/T203630) (owner: 10Legoktm) [05:51:18] (03PS2) 10Legoktm: Add experimental mediawiki-core-php70-phan-seccheck-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/462341 (https://phabricator.wikimedia.org/T203630) [05:51:20] (03PS1) 10Legoktm: Factor out docker-ci-src-setup-mw macro [integration/config] - 10https://gerrit.wikimedia.org/r/462342 [05:56:30] Project beta-scap-eqiad build #222885: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222885/ [05:58:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [06:05:54] (03PS2) 10Legoktm: Factor out docker-ci-src-setup-mw macro [integration/config] - 10https://gerrit.wikimedia.org/r/462342 [06:05:56] (03PS3) 10Legoktm: Add experimental mediawiki-core-php70-phan-seccheck-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/462341 (https://phabricator.wikimedia.org/T203630) [06:06:18] (03CR) 10Legoktm: [C: 032] Factor out docker-ci-src-setup-mw macro [integration/config] - 10https://gerrit.wikimedia.org/r/462342 (owner: 10Legoktm) [06:06:50] (03CR) 10Legoktm: [C: 032] Add experimental mediawiki-core-php70-phan-seccheck-docker job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/462341 (https://phabricator.wikimedia.org/T203630) (owner: 10Legoktm) [06:08:13] (03Merged) 10jenkins-bot: Factor out docker-ci-src-setup-mw macro [integration/config] - 10https://gerrit.wikimedia.org/r/462342 (owner: 10Legoktm) [06:10:07] Project beta-scap-eqiad build #222886: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222886/ [06:10:50] (03Merged) 10jenkins-bot: Add experimental mediawiki-core-php70-phan-seccheck-docker job [integration/config] - 10https://gerrit.wikimedia.org/r/462341 (https://phabricator.wikimedia.org/T203630) (owner: 10Legoktm) [06:23:26] Project beta-scap-eqiad build #222887: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222887/ [06:36:43] Project beta-scap-eqiad build #222888: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222888/ [06:41:03] !log deploying https://gerrit.wikimedia.org/r/462341 [06:41:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:49:51] Project beta-scap-eqiad build #222889: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222889/ [06:51:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimedia-Logstash: logstash-beta.wmflab throws multiple "Error: Could not locate that visualization" - https://phabricator.wikimedia.org/T204845 (10fgiunchedi) a:05fgiunchedi>03None Thanks @greg that's ind... [07:02:52] Project beta-scap-eqiad build #222890: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222890/ [07:15:59] Project beta-scap-eqiad build #222891: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222891/ [07:24:17] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/wikinity] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462389 [07:24:20] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/wikinity] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462389 (owner: 10QChris) [07:24:51] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/wikinity] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462390 [07:24:54] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/wikinity] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462390 (owner: 10QChris) [07:26:25] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [07:27:36] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/commons-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462394 [07:27:39] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/commons-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462394 (owner: 10QChris) [07:28:00] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/commons-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462395 [07:28:03] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/commons-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462395 (owner: 10QChris) [07:29:00] Project beta-scap-eqiad build #222892: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222892/ [07:30:53] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/weapon-of-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462396 [07:30:55] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/weapon-of-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462396 (owner: 10QChris) [07:31:11] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/weapon-of-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462397 [07:31:13] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/weapon-of-mass-description] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462397 (owner: 10QChris) [07:31:44] 10MediaWiki-Releasing, 10Security: Streamline MW security release process - https://phabricator.wikimedia.org/T196602 (10Legoktm) I think the most relevant comment/suggestion to go forwards is still T156445#3088982. [07:32:52] 10MediaWiki-Releasing: Move to a `git archive` like model for MediaWiki tarball creation - https://phabricator.wikimedia.org/T199467 (10Legoktm) 05Open>03Resolved I wrote up https://blog.legoktm.com/2018/09/24/writing-a-new-mediawiki-tarball-release-script.html going into more detail about this project. [07:33:15] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/harvesting-data-refinery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462398 [07:33:18] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/harvesting-data-refinery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462398 (owner: 10QChris) [07:33:34] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/harvesting-data-refinery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462399 [07:33:37] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/harvesting-data-refinery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462399 (owner: 10QChris) [07:39:35] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [07:42:43] Project beta-scap-eqiad build #222893: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222893/ [07:55:59] Project beta-scap-eqiad build #222894: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222894/ [08:09:07] Project beta-scap-eqiad build #222895: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222895/ [08:18:15] (03CR) 10KartikMistry: [C: 031] "@hashar, any update on this? Any blocker?" [integration/config] - 10https://gerrit.wikimedia.org/r/450508 (https://phabricator.wikimedia.org/T86930) (owner: 10Santhosh) [08:22:34] Project beta-scap-eqiad build #222896: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222896/ [08:32:33] !log deployment-deploy01 rm -fR /tmp/scap_l10n_* [08:32:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:39:04] Yippee, build fixed! [08:39:05] Project beta-scap-eqiad build #222897: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222897/ [08:43:19] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [08:44:28] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T191069 (10zeljkofilipin) a:03zeljkofilipin [08:44:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T191069 (10zeljkofilipin) [08:55:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) I gave it a quick try on deployment-deploy01 reproducing `scap.tasks.cache_git_info()`. Namely getting... [09:11:07] Project beta-scap-eqiad build #222900: 15ABORTED in 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222900/ [09:13:12] (03PS1) 10MarcoAurelio: Edit Project Config [extensions/DateDiff] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462410 [09:13:36] (03PS2) 10MarcoAurelio: Edit Project Config [extensions/DateDiff] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462410 [09:13:45] (03CR) 10MarcoAurelio: [V: 032 C: 032] Edit Project Config [extensions/DateDiff] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462410 (owner: 10MarcoAurelio) [09:20:01] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) I ended up manually hacking scap on `deployment-deploy01` https://integration.wikimedia.org/ci/job/beta... [09:24:08] !log Live hacked scap code on deployment-deploy01 for T204762 and reverted hack changes [09:24:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:24:15] T204762: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 [09:50:25] Project beta-scap-eqiad build #222903: 15ABORTED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222903/ [09:51:13] 10Phabricator: Mass-edits via @Phabricator_maintenance account stop after 11 tasks - https://phabricator.wikimedia.org/T205258 (10Aklapper) p:05Triage>03Low [09:51:48] !log deployment-deploy01 : backed up /srv/mediawiki-staging/php-master/cache/gitinfo and created a new. Its size of 69632 bytes might cause slow writes?? | T204762 [09:51:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:51:54] T204762: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 [09:56:28] 10Phabricator: Mass-edits via @Phabricator_maintenance account stop after 11 tasks - https://phabricator.wikimedia.org/T205258 (10Aklapper) I had checked that the account is a member of #Trusted-Contributors. I know once had 10 instead of 11 tasks edited in one batch, so it's likely a time limit being hit. [10:39:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10hashar) The spam of `close(x) = -1 EBADF (Bad file descriptor)` comes from a call to `os.closerange` ``` lang=p... [10:47:46] 10Continuous-Integration-Config, 10Gerrit, 10Tools, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) [10:52:23] (03PS1) 10Hashar: Remove jenkins-mwext-sync group [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462419 [10:52:29] (03CR) 10Hashar: [V: 032 C: 032] Remove jenkins-mwext-sync group [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462419 (owner: 10Hashar) [10:53:49] (03PS1) 10Hashar: Modify access rules [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462420 [10:54:30] (03PS2) 10Hashar: Allow project owners to submit [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462420 [10:55:11] !log gerrit: granting labs/tools/* project owners the ability to submit changes | https://gerrit.wikimedia.org/r/#/c/labs/tools/+/462420/ [10:55:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:55:18] (03CR) 10Hashar: [V: 032 C: 032] Allow project owners to submit [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/462420 (owner: 10Hashar) [10:56:10] 10Continuous-Integration-Config, 10Gerrit, 10Tools, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10hashar) Meanwhile, in Gerrit I have granted permissions to project owners to submit changes ( https://gerrit.wikimedia.org... [11:11:24] (03PS1) 10Urbanecm: Add CI config for Commons Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462429 [11:11:26] (03PS1) 10Urbanecm: Add CI config for Weapon of Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462430 [11:11:28] (03PS1) 10Urbanecm: Add CI config for Harvesting Data Refinery [integration/config] - 10https://gerrit.wikimedia.org/r/462431 [11:11:30] (03PS1) 10Urbanecm: Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 [11:14:07] (03CR) 10jerkins-bot: [V: 04-1] Add CI config for Harvesting Data Refinery [integration/config] - 10https://gerrit.wikimedia.org/r/462431 (owner: 10Urbanecm) [11:14:24] (03CR) 10jerkins-bot: [V: 04-1] Add CI config for Weapon of Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462430 (owner: 10Urbanecm) [11:14:52] (03CR) 10jerkins-bot: [V: 04-1] Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 (owner: 10Urbanecm) [11:16:07] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10MoritzMuehlenhoff) Compared to the rest, mwdebug* are VMs, how large is the difference to the other servers you were seeing? [11:17:13] (03PS2) 10Urbanecm: Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 [11:19:32] (03PS2) 10Urbanecm: Add CI config for Weapon of Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462430 [11:19:35] (03CR) 10Hashar: [C: 032] Add CI config for Commons Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462429 (owner: 10Urbanecm) [11:21:16] (03CR) 10jerkins-bot: [V: 04-1] Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 (owner: 10Urbanecm) [11:21:38] (03Merged) 10jenkins-bot: Add CI config for Commons Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462429 (owner: 10Urbanecm) [11:21:58] (03CR) 10jerkins-bot: [V: 04-1] Add CI config for Weapon of Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462430 (owner: 10Urbanecm) [11:24:55] (03PS3) 10Hashar: Add CI config for Weapon of Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462430 (owner: 10Urbanecm) [11:27:56] (03PS3) 10Urbanecm: Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 [11:28:05] (03CR) 10Hashar: [C: 032] Add CI config for Weapon of Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462430 (owner: 10Urbanecm) [11:28:17] (03PS2) 10Hashar: Add CI config for Harvesting Data Refinery [integration/config] - 10https://gerrit.wikimedia.org/r/462431 (owner: 10Urbanecm) [11:28:31] (03PS4) 10Hashar: Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 (owner: 10Urbanecm) [11:32:14] (03Merged) 10jenkins-bot: Add CI config for Weapon of Mass Description [integration/config] - 10https://gerrit.wikimedia.org/r/462430 (owner: 10Urbanecm) [11:32:16] (03CR) 10Hashar: [C: 032] Add CI config for Harvesting Data Refinery [integration/config] - 10https://gerrit.wikimedia.org/r/462431 (owner: 10Urbanecm) [11:32:24] (03CR) 10Hashar: [C: 032] Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 (owner: 10Urbanecm) [11:34:28] 10Continuous-Integration-Config, 10Gerrit, 10Tools, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10Urbanecm) [11:35:56] (03Merged) 10jenkins-bot: Add CI config for Harvesting Data Refinery [integration/config] - 10https://gerrit.wikimedia.org/r/462431 (owner: 10Urbanecm) [11:35:58] (03Merged) 10jenkins-bot: Add CI config for Wikinity [integration/config] - 10https://gerrit.wikimedia.org/r/462432 (owner: 10Urbanecm) [11:37:32] 10Continuous-Integration-Config, 10Gerrit, 10Tools, 10Patch-For-Review: Add CI to all labs/tools/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180318 (10Urbanecm) [11:41:27] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [11:43:08] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [11:48:35] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [12:47:16] (03CR) 10Alex Monk: [C: 04-1] "doesn't do anything" [debs/prometheus-openstack-exporter] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/461382 (owner: 10Arturo Borrero Gonzalez) [12:52:09] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) Thanks @hashar, [[ https://wikimedia.biterg.io | Bitergia ]] is really useful! Their report for the last year:... [12:56:49] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) `git effort` results for `mediawiki/extensions`. {P7585} [12:58:03] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [13:07:10] https://github.com/auchenberg/volkswagen [13:07:13] rofl [13:15:39] lolol [13:24:47] lol [13:25:21] cease and desist in 5...4... [13:53:29] (03PS1) 10Hashar: Archive marvin [integration/config] - 10https://gerrit.wikimedia.org/r/462473 (https://phabricator.wikimedia.org/T205170) [13:53:45] (03CR) 10Hashar: [C: 032] Archive marvin [integration/config] - 10https://gerrit.wikimedia.org/r/462473 (https://phabricator.wikimedia.org/T205170) (owner: 10Hashar) [13:56:13] (03Merged) 10jenkins-bot: Archive marvin [integration/config] - 10https://gerrit.wikimedia.org/r/462473 (https://phabricator.wikimedia.org/T205170) (owner: 10Hashar) [14:48:07] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:51:23] 10Project-Admins: Create a component project for the EditAccount extension - https://phabricator.wikimedia.org/T205293 (10planetenxin) [15:01:45] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:21:25] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:41:34] 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865 (10Motionlab) I'm really struggling with this. Is this an auto-magic field? In that I simply add a field to Maniphest and it's supposed to work? my Maniphest.custom-field-definitions loo... [15:52:52] hashar: o/ [15:53:30] elukey: yup? [15:53:41] have you ever got something like 'fatal: Couldn't JIT the PCRE2 pattern '(? (bundle install works, ruby 2.3.3) [15:54:03] nop that does not ring a bell [15:55:18] ack thanks :) [15:55:28] elukey: which change? [15:56:03] (03PS2) 10Hashar: Decouple composer/npm run for extskin [integration/quibble] - 10https://gerrit.wikimedia.org/r/461998 [15:56:05] (03PS2) 10Hashar: Early return in extskin composer/npm test commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/461999 [15:56:07] (03PS1) 10Hashar: Parallelize ext/skin linter [integration/quibble] - 10https://gerrit.wikimedia.org/r/462512 [15:56:09] (03PS1) 10Hashar: Parallelize mediawiki/core linter [integration/quibble] - 10https://gerrit.wikimedia.org/r/462513 [15:56:19] it does that even with the current ops repo, so I have probably something misconfigured [15:56:20] * hashar all those quibble changes are terrible :^ [15:56:41] elukey: OHHH [15:56:55] elukey: so that regex comes from the /typos file [15:57:08] it is passed to git grep [15:57:08] rake_modules/taskgen.rb: system("git grep -I -n -P -f typos -- #{shell_files}") [15:57:21] so probably your git/pcre has some issue :\ [15:57:34] note the -P which instructs "git grep" to use pcre regex [15:58:28] ahhh okok thanks for the hint! [15:58:31] so in theory, you should reproduce it from the root of the repo [15:58:33] using git grep -I -n -P '(? elukey: or in short, I am tempted to blame libpcre :] [15:59:53] 10Release-Engineering-Team (Kanban), 10Scap: scap clean not cleaning up fully - https://phabricator.wikimedia.org/T205002 (10thcipriani) p:05Triage>03Normal That'll happen when you do a `--delete`, i.e. the branch directory will sit there empty until then next `scap sync` is run. This is happening because... [16:00:38] hashar: will investigate, thanks a lot! [16:01:37] 10Release-Engineering-Team (Kanban), 10Scap: scap clean not cleaning up fully - https://phabricator.wikimedia.org/T205002 (10Reedy) Those directories were still in `/srv/mediawiki-staging` on the deployment host, hence reporting the bug based on the investigation for {T205000} [16:06:18] 10Phabricator: Mass-edits via @Phabricator_maintenance account stop after 11 tasks - https://phabricator.wikimedia.org/T205258 (10mmodell) That's odd :-/ [16:21:31] Project beta-scap-eqiad build #222930: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222930/ [16:21:32] Project beta-update-databases-eqiad build #28559: 04FAILURE in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28559/ [16:22:24] Project beta-scap-eqiad build #222931: 04STILL FAILING in 4.7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222931/ [16:37:11] Yippee, build fixed! [16:37:12] Project beta-scap-eqiad build #222932: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222932/ [16:40:22] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Developer-Advocacy, 10GitHub-Mirrors, and 2 others: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540 (10hashar) a:05hashar>03None [16:41:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-production-error (Shared Build Failure): Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge - https://phabricator.wikimedia.org/T179963 (10hashar) a:05hashar>03None [16:42:27] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Quibble, 10Patch-For-Review: Quibble may need to rebuild localization cache before running tests - https://phabricator.wikimedia.org/T196347 (10hashar) Pending on #mediawiki-installer patch https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/44... [16:44:40] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Kubernetes: Draft requirements for a Kubernetes based solution for CI - https://phabricator.wikimedia.org/T183513 (10hashar) a:05hashar>03None [16:51:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10DBA, 10MediaWiki-Database, 10Quibble: Enable MariaDB/MySQL strict mode on CI db hosts - https://phabricator.wikimedia.org/T119371 (10hashar) a:05hashar>03None For now CI has `sql_mode = 'TRADITIONAL'` unfortunately I lack... [16:53:51] 10Release-Engineering-Team (Kanban), 10ORES, 10Quibble, 10Scoring-platform-team, and 5 others: Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server) - https://phabricator.wikimedia.org/T198201 (10hashar) a:05hashar>03None The t... [17:15:50] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [17:15:52] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Document which per-patch/daily Jenkins job is running for repositories with Ruby/Node.js Selenium tests - https://phabricator.wikimedia.org/T198412 (10zeljkofilipin) 05Open>03declined I don't remember why this was needed f... [17:16:55] 10Release-Engineering-Team (Someday), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Someday/maybe Selenium framework improvements - https://phabricator.wikimedia.org/T190995 (10zeljkofilipin) [17:16:57] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Documentation, 10JavaScript, 10User-zeljkofilipin: Blog posts about new Selenium framework features - https://phabricator.wikimedia.org/T191982 (10zeljkofilipin) [17:17:41] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [17:18:30] 10Release-Engineering-Team (Someday), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Someday/maybe Selenium framework improvements - https://phabricator.wikimedia.org/T190995 (10zeljkofilipin) [17:18:32] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [17:18:38] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Selenium tests should be easier to run - https://phabricator.wikimedia.org/T182691 (10zeljkofilipin) [17:18:48] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [17:20:43] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [17:22:21] Yippee, build fixed! [17:22:22] Project beta-update-databases-eqiad build #28560: 09FIXED in 2 min 20 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28560/ [17:28:31] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10Legoktm) IIRC it was about 3-4 minutes. But that was also with HHVM, it's probably worth checking again with PHP 7 to see if... [17:29:37] 10Continuous-Integration-Config, 10MinervaNeue, 10phan-taint-check-plugin, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Minerva taint error: Calling method \BaseTemplate::set() in \SkinMinerva::prepareHeaderAndFooter that outputs using tainted argument - https://phabricator.wikimedia.org/T204094 (... [17:29:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-production-error (Shared Build Failure): Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge - https://phabricator.wikimedia.org/T179963 (10Seb35) `git-changed-in-head` works with almost-shallow clones:... [17:30:22] 10Release-Engineering-Team, 10Scap: l10nupdate is still using HHVM - https://phabricator.wikimedia.org/T205313 (10Legoktm) [17:32:20] 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865 (10Aklapper) @motionlab: Please ask general Phabricator questions which are not directly related to Wikimedia's Phabricator instance in the Phabricator forum at https://discourse.phabricat... [17:34:51] 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865 (10Motionlab) will do, apologies [17:44:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-production-error (Shared Build Failure): Workspaces for mwgate-php55lint / mwgate-php70lint are getting huge - https://phabricator.wikimedia.org/T179963 (10hashar) Yup that works most of the time, but --depth 2 is not s... [17:47:51] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Document which per-patch/daily Jenkins job is running for repositories with Ruby/Node.js Selenium tests - https://phabricator.wikimedia.org/T198412 (10hashar) I think that followed one of our 1/1 discussion with me being confu... [17:49:55] 10Release-Engineering-Team (Kanban), 10Scap: scap clean not cleaning up fully - https://phabricator.wikimedia.org/T205002 (10thcipriani) >>! In T205002#4611234, @Reedy wrote: > Those directories were still in `/srv/mediawiki-staging` on the deployment host, hence reporting the bug based on the investigation fo... [17:51:47] 10Release-Engineering-Team (Kanban), 10Scap: scap clean not cleaning up fully - https://phabricator.wikimedia.org/T205002 (10Reedy) I think so, yeah. Didn't actually check the targets at the time, but the folders are definitely gone from `/srv/mediawiki` now too [17:57:43] 10Release-Engineering-Team (Kanban), 10Scap: scap clean not cleaning up deployment host fully - https://phabricator.wikimedia.org/T205002 (10thcipriani) [18:06:20] 10Project-Admins: Create a component project for the EditAccount extension - https://phabricator.wikimedia.org/T205293 (10Aklapper) @planetenxin: Hi, are you a (co-)maintainer of that extension, and/or did the maintainers agree with this? We do not want to create [more] feedback places which would give bug repo... [18:13:25] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1): exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Jdlrobson) Any luck @Ottoma... [18:28:58] 10Release-Engineering-Team, 10Scap: l10nupdate is still using HHVM - https://phabricator.wikimedia.org/T205313 (10Aklapper) (Is this a subtask of T176370?) [18:56:04] 10Release-Engineering-Team, 10Scap: l10nupdate is still using HHVM - https://phabricator.wikimedia.org/T205313 (10Jdforrester-WMF) Properly it should have been a sub-task of T195393 but… ;-) [19:29:00] (03PS1) 10Legoktm: Upgrade squizlabs/php_codesniffer to 3.3.2 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/462562 [20:00:08] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Puppet, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1): exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Ottoma... [20:01:59] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Ottomata) I think ^ is what is needed (sorry wa... [20:21:54] !log (beta): Update mobileapps to badb463 [20:21:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:58:18] (03CR) 10Reedy: [C: 032] Upgrade squizlabs/php_codesniffer to 3.3.2 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/462562 (owner: 10Legoktm) [20:59:53] (03Merged) 10jenkins-bot: Upgrade squizlabs/php_codesniffer to 3.3.2 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/462562 (owner: 10Legoktm) [21:00:31] (03CR) 10jenkins-bot: Upgrade squizlabs/php_codesniffer to 3.3.2 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/462562 (owner: 10Legoktm) [21:30:35] 10Project-Admins: Replace tracking bug T21719 by new project tag "HTML5" - https://phabricator.wikimedia.org/T102502 (10MarcoAurelio) After some years without any comments, I move that this is approved and a HTML-5 tag is created. [21:40:13] 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Discovery: Deploy wikimedia/portals with scap3 - https://phabricator.wikimedia.org/T114694 (10MarcoAurelio) Currently weekly portal deploys happen thanks to @Jdrewniak - how's the current process? Thanks. [21:41:40] 10Gerrit: Deploy gerrit master (then 2.16) to gerrit.git.wmflabs.org - https://phabricator.wikimedia.org/T205346 (10Paladox) p:05Triage>03Lowest [21:52:46] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T191069 (10Jdlrobson) [21:52:48] marxarelli: are you still looking into the performance/runtime of VMs? https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php70-docker/751/consoleFull looks like a huge anomaly in the amount of time it took to run the PHPUnit tests [22:40:55] (03CR) 10Jforrester: [C: 032] Order list of extensions by alphabet [tools/release] - 10https://gerrit.wikimedia.org/r/455186 (owner: 10Aklapper) [22:42:04] (03Merged) 10jenkins-bot: Order list of extensions by alphabet [tools/release] - 10https://gerrit.wikimedia.org/r/455186 (owner: 10Aklapper) [22:43:32] (03CR) 10Jforrester: [C: 032] Stop branching DisableAccount [tools/release] - 10https://gerrit.wikimedia.org/r/461733 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:43:40] (03PS2) 10Jforrester: Stop branching DisableAccount [tools/release] - 10https://gerrit.wikimedia.org/r/461733 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:46:24] legoktm: thanks [22:46:29] wow, 45 min. that's really slow [22:46:43] yeaaaahhh :(( [22:46:54] i haven't seen any in the data sets recently that took that long [22:47:14] although the m1.medium haven't been performing with consistently more variance than the other types [22:47:23] *have* been [22:47:42] most recent report is here https://docs.google.com/spreadsheets/d/1tbQk0P-Ih2RLS8MZ5-uBGCeWGq3Aig-IiIkzXLfLoYU/edit?usp=sharing [22:49:27] oh, interesting. i think i know why they haven't been in the data sets. my reports aren't taking in account failures due to build timeouts [22:49:46] they only look at successful builds