[02:43:41] Project beta-scap-eqiad build #204957: 04FAILURE in 0.42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204957/ [02:59:10] Yippee, build fixed! [02:59:11] Project beta-scap-eqiad build #204958: 09FIXED in 5 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/204958/ [06:56:58] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4148707 (10Joe) >>! In T191921#4147196, @Krinkle wrote: > @thcipriani Hm.. these are seconds though, as opposed to minutes. Is there something differen... [07:01:29] 10Phabricator: Email sometimes not being sent when a task is created and "other task activity" is not set in user preferences - https://phabricator.wikimedia.org/T182549#4148710 (10mmodell) @EddieGP thanks for summarizing the situation. I'm not sure if a custom hack is warranted either but I will take a look at... [07:27:47] !log Upgrading Blue Ocean on CI Jenkins 1.4.2 -> 1.5.0 [07:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:33:01] Project mwext-phpunit-coverage-publish build #3671: 04FAILURE in 3.7 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3671/ [09:33:04] Project mwext-phpunit-coverage-publish build #3672: 04STILL FAILING in 2.9 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3672/ [09:33:07] Project mwext-phpunit-coverage-publish build #3673: 04STILL FAILING in 2 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3673/ [09:35:28] Yippee, build fixed! [09:35:29] Project mwext-phpunit-coverage-publish build #3674: 09FIXED in 2 min 15 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3674/ [12:00:44] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Failed loading configuration file ... wdio.conf.jenkins.js - https://phabricator.wikimedia.org/T192367#4149426 (10zeljkofilipin) 05Open>03Resolved [12:07:49] (03PS1) 10Hashar: Migrate Vector skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/428323 [12:09:06] (03CR) 10jerkins-bot: [V: 04-1] Migrate Vector skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/428323 (owner: 10Hashar) [12:11:27] (03PS2) 10Hashar: Migrate Vector skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/428323 [12:14:44] (03PS3) 10Hashar: Migrate Vector skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/428323 [12:14:46] (03PS1) 10Hashar: Ensure quibble jobs are in the same queue [integration/config] - 10https://gerrit.wikimedia.org/r/428325 [12:21:14] (03CR) 10Hashar: [C: 032] Ensure quibble jobs are in the same queue [integration/config] - 10https://gerrit.wikimedia.org/r/428325 (owner: 10Hashar) [12:22:13] (03CR) 10Hashar: [C: 032] Migrate Vector skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/428323 (owner: 10Hashar) [12:22:32] (03Merged) 10jenkins-bot: Ensure quibble jobs are in the same queue [integration/config] - 10https://gerrit.wikimedia.org/r/428325 (owner: 10Hashar) [12:23:54] (03Merged) 10jenkins-bot: Migrate Vector skin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/428323 (owner: 10Hashar) [12:36:29] hmm. [12:37:40] hashar: https://gerrit.wikimedia.org/r/#/c/428146/ am i interpreting this correctly ? is qunit not triggered for that patchset ? is that due to the quibble stuff ? [12:42:27] thedj: yup. One command "quibble" to run them all [12:42:39] thedj: the aim is to make it easy for developers to reproduce a CI run locally [12:43:33] thedj: here is a run for Vector https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/143/consoleFull [12:43:52] thedj: it runs composer test / npm test, the mediawiki phpunit testsuite "skin", qunit tests etc [12:47:26] hashar: but i don't see quibble triggered for that patchset. [12:47:44] i'm just not sure if the testcases actually ran and if it is safe to merge..... [12:55:39] 10Continuous-Integration-Config, 10TemplateData, 10VisualEditor: Set up phpunit coverage job for TemplateData - https://phabricator.wikimedia.org/T192775#4149591 (10Krinkle) [12:55:51] 10Continuous-Integration-Config, 10TemplateData, 10VisualEditor: Set up phpunit coverage job for TemplateData - https://phabricator.wikimedia.org/T192775#4149601 (10Krinkle) [12:55:51] thedj: ah sorry [12:56:09] thedj: that is because MatthiasDD is not in the whitelist [12:56:23] thedj: you gotta comment "recheck" in Gerrit to have the tests to be triggered [12:56:30] eventually we will get rid of that whitelist [12:56:50] 10Continuous-Integration-Config, 10TemplateData, 10VisualEditor: Set up phpunit coverage job for TemplateData - https://phabricator.wikimedia.org/T192775#4149591 (10Krinkle) [13:05:16] hashar: ah got it. [13:21:13] hashar: Could use your help on the quibble/selenium patch. Not sure what to do with ChromiumDriver there [13:21:30] https://gerrit.wikimedia.org/r/#/c/428040/ [13:21:58] 'npm run selenium' runs https://github.com/wikimedia/mediawiki/blob/master/tests/selenium/selenium.sh - if we use 'npm run selenium' in quibble, do we still need the python layer for ChromiumDriver? [13:23:33] Krinkle: in ~ 40 minutes. I am in a call right now :) [13:25:16] k :) [13:33:48] 10MediaWiki-Releasing, 10MediaWiki-extensions-ReplaceText, 10MW-1.31-release: Bundle Replace Text extension with MW 1.31 - https://phabricator.wikimedia.org/T191741#4115157 (10Bawolff) For SQLite, I found https://stackoverflow.com/questions/4663368/custom-regexp-function-to-be-used-in-a-sqlite-select-stateme... [14:30:30] Krinkle: sorry took longer than expected [14:30:43] Krinkle: actually I talked about that with zeljkof this morning [14:30:53] I went with Quibble managing xvfb/display/chromedriver directly [14:31:07] so it has all the logic and end up just running the test command [14:32:04] essentially, quibble ignores npm run selenium and the tests/selenium/selenium.sh script. PArtly because they got introduced after I had it implemented in quibble [14:32:47] i will reply on the change [14:35:31] hashar: OK. I don't know if we want to keep that (maybe worth to keep if it runs faster this way, or does better logging?), but logically it seems preferably for this logic to exist in mediawiki/core and not individual jobs, for maintenance. But if we want to keep the logic in quibble, perhaps we can create a second 'npm run' command that runs just the tests. [14:35:39] e.g. 'npm run selenium-internal' [14:35:47] That way, at least the grunt command is not hardcoded. [14:35:58] So it can be changed and improved, like I'm doing at the moment, which fails because of quibble. [14:38:55] Krinkle: I think we can keep it in both [14:39:12] tests/selenium/selenium.sh | npm run selenium being conveniences helpers for developers [14:40:30] hashar: Okay, that seems fine, but I'm just not sure why. What is the benefit of quibble doing this directly? Would it not be easier to maintain if this was just central in selenium.sh? [14:41:11] Also, would it be fine to at least abstract the grunt command into a separate npm-run command so that at least the configuration and entry point can be changed (which are more likely to change than the chromedriver, which is fine and won't change often) [14:41:28] See https://gerrit.wikimedia.org/r/#/c/428039/ for example of a failure [14:42:05] I would imagine you like to move this into core to keep jobs simpler (e.g. more like .travis.yml etc., logic in the repo, not in CI) [14:42:30] also with versioning between branches etc. Gonna get quibble complicated when it differs between branches [14:48:54] (03CR) 10Hashar: "The "npm run selenium" had a few issues:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/428040 (https://phabricator.wikimedia.org/T179190) (owner: 10Krinkle) [14:49:55] Krinkle: yes zeljkof added the selenium.sh to abstract things out of CI / quibble / jenkins etc [14:50:15] Sounds like it should be fixed to not do this exit code thing. I think that is achievable. [14:50:36] It is also achievable to not kill the process by name, but use PID instead. [14:50:55] yup [14:51:32] but it is easier to do in python rather than in shell in multiple mediawiki branches [14:51:39] OK. I'll leave that to you guys to figure out :) [14:51:46] I'll leave this for now, given that you're working on it. [14:51:55] But I will submit a patch to change just the grunt part of it [14:52:01] and leave chromiumdriver/exitcode unchagned. [14:52:05] Sounds good? [14:52:35] Krinkle: https://gerrit.wikimedia.org/r/#/c/428039/1/tests/selenium/selenium.sh ? [14:53:36] hashar: Yeah, we can either 1) update quibble to do the same (hard code) and keep duplication of the test command until it is selenium.sh is improved, or 2) Create a new command 'npm run selenium-test' that has this 1 line in it, and use that from quibble. [14:54:41] Krinkle: 2) ! [14:54:54] and make quibble as simple as possible [14:54:58] OK! [14:57:23] 10Release-Engineering-Team (Kanban), 10MW-1.31-release: Release MW 1.31 - https://phabricator.wikimedia.org/T191088#4150038 (10Dereckson) [14:58:08] 10Release-Engineering-Team (Kanban), 10MW-1.31-release: Release MW 1.31 - https://phabricator.wikimedia.org/T191088#4093029 (10Dereckson) Release notes are currently including a 1.32 change entry for 1.31: T189127. [15:15:33] 10MediaWiki-Releasing, 10MediaWiki-extensions-CategoryTree, 10MW-1.31-release: Bundle CategoryTree extension with MW 1.31 - https://phabricator.wikimedia.org/T191735#4115103 (10Jdforrester-WMF) Actual patch to do this is https://gerrit.wikimedia.org/r/#/c/427331/ BTW. [15:26:31] 10Scap: sync-l10n doesn't inject some parameters in the commands - https://phabricator.wikimedia.org/T192801#4150129 (10Dereckson) [15:42:46] (03PS1) 10Krinkle: Set MW_SCRIPT_PATH to '/' instead of '' [integration/quibble] - 10https://gerrit.wikimedia.org/r/428356 [15:43:43] (03PS2) 10Krinkle: Set MW_SCRIPT_PATH to '/' instead of '' [integration/quibble] - 10https://gerrit.wikimedia.org/r/428356 [16:24:11] 10Release-Engineering-Team (Kanban): Investigate the Extension Submittal Review process - https://phabricator.wikimedia.org/T182731#4150486 (10greg) p:05Triage>03Low [16:26:34] 10Phabricator, 10Release-Engineering-Team (Kanban), 10media-storage: Connect Phabricator to swift for storage of git-lfs and file uploads. - https://phabricator.wikimedia.org/T182085#4150489 (10mmodell) I finally made some good progress on the code last week. I should have something ready to play with this w... [16:27:20] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Vagrant, 10User-zeljkofilipin: Create a test suite that compiles mediawiki-vagrant puppet manifests - https://phabricator.wikimedia.org/T183570#4150492 (10greg) p:05Low>03Lowest De-emphasizing due to concerns over usability of this. [16:27:44] 10Release-Engineering-Team (Kanban), 10Wikimedia-Blog-Content: [Technical Debt Series]What is Technical Debt - https://phabricator.wikimedia.org/T175181#4150495 (10greg) 05Open>03Resolved [16:29:25] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#4150505 (10mmodell) a:05mmodell>03None [16:30:09] 10Phabricator, 10Release-Engineering-Team (Kanban): Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393#4150510 (10mmodell) a:05mmodell>03None [16:31:05] 10Differential, 10Phabricator, 10Wikibugs, 10Gerrit-Migration, 10WorkType-NewFunctionality: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417#4150514 (10mmodell) [16:32:20] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Use git as transport mechanism for MediaWiki scap deploys - https://phabricator.wikimedia.org/T147938#4150519 (10mmodell) [16:32:54] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install helm on integration agents - https://phabricator.wikimedia.org/T188934#4150525 (10thcipriani) [16:33:08] 10Release-Engineering-Team (Kanban), 10Scap: Add jobrunners to Scap canary process - https://phabricator.wikimedia.org/T172480#4150530 (10mmodell) [16:34:06] 10Release-Engineering-Team (Watching / External), 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#4150531 (10greg) [16:38:49] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations: mwscript rebuildLocalisationCache.php takes 40 minutes - https://phabricator.wikimedia.org/T191921#4150546 (10thcipriani) using `Eval.Jit=1` does actually seem faster on deployment-tin, anyway ``` [thcipriani@deployment-tin ~]$ export PHP='hhvm -vEva... [16:41:50] 10Release-Engineering-Team (Kanban), 10Analytics-Tech-community-metrics, 10Code-Health: Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891#4150561 (10greg) p:05Triage>03Normal [16:42:32] 10Phabricator, 10Release-Engineering-Team (Kanban): Prepare a disaster recovery plan for failing over from phab1001 to phab2001 (or phab2001 to 1001) - https://phabricator.wikimedia.org/T190572#4150566 (10greg) [16:45:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10LDAP-Access-Requests, and 4 others: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811#4150582 (10greg) [16:45:11] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10LDAP-Access-Requests, and 4 others: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811#4054206 (10greg) RelEng related things here done, the rest are in Ops bailiwick (mailing list and ld... [16:48:12] 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Follow up with platform team regarding refactoring plans - https://phabricator.wikimedia.org/T189062#4150612 (10greg) [16:48:45] 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Follow up with platform team regarding refactoring plans - https://phabricator.wikimedia.org/T189062#4029507 (10greg) p:05Triage>03Normal [16:49:33] 10Release-Engineering-Team (Kanban), 10MediaWiki-Configuration, 10Wikimedia-Incident: Find out if there's a plan to get rid of using globals in PHP code - https://phabricator.wikimedia.org/T189059#4150616 (10greg) [16:50:42] 10Release-Engineering-Team (Someday), 10MediaWiki-extensions-PoolCounter, 10Patch-For-Review, 10Wikimedia-log-errors (Jenkins Failure): Fix tests of PoolCounter extension - https://phabricator.wikimedia.org/T178517#4150618 (10greg) [16:51:11] 10Release-Engineering-Team (Someday), 10MediaWiki-extensions-PoolCounter, 10Patch-For-Review, 10Wikimedia-log-errors (Jenkins Failure): Fix tests of PoolCounter extension - https://phabricator.wikimedia.org/T178517#3694596 (10greg) FTR: The entry for steward/maintainer for poolcounter is empty on [[ https:... [16:53:06] 10Release-Engineering-Team (Kanban), 10User-greg: Setup Learning Circles for RelEng - https://phabricator.wikimedia.org/T188706#4150624 (10greg) p:05Low>03Normal [16:54:13] 10Release-Engineering-Team (Kanban), 10User-greg: Create peer feedback form - https://phabricator.wikimedia.org/T188703#4150633 (10greg) 05Open>03declined waited too long, let's just wait until the normal/big one [16:58:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-SWAT-deployments, 10User-zeljkofilipin: Proposal: Effective immediately, disallow multi-sync patch deployment - https://phabricator.wikimedia.org/T187761#4150680 (10greg) p:05Triage>03Normal +1 let's document this... [17:03:20] 10Release-Engineering-Team (Next), 10MediaWiki-Configuration, 10Wikimedia-Incident: Find out if there's a plan to get rid of using globals in PHP code - https://phabricator.wikimedia.org/T189059#4150704 (10greg) [17:04:00] 10Release-Engineering-Team (Watching / External), 10MediaWiki-extensions-PoolCounter, 10Patch-For-Review, 10Wikimedia-log-errors (Jenkins Failure): Fix tests of PoolCounter extension - https://phabricator.wikimedia.org/T178517#4150707 (10greg) [17:05:48] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Documentation: Prepare a disaster recovery plan for failing over from phab1001 to phab2001 (or phab2001 to 1001) - https://phabricator.wikimedia.org/T190572#4150727 (10greg) [17:09:11] 10Scap: sync-l10n doesn't inject some parameters in the commands - https://phabricator.wikimedia.org/T192801#4150736 (10Dereckson) 05Open>03Resolved a:03thcipriani [17:37:58] (03PS2) 10Krinkle: Run `npm run selenium` instead of `grunt webdriver:test` [integration/quibble] - 10https://gerrit.wikimedia.org/r/428040 (https://phabricator.wikimedia.org/T179190) [17:38:39] (03PS3) 10Krinkle: Run `npm run selenium-test` instead of `grunt webdriver:test` [integration/quibble] - 10https://gerrit.wikimedia.org/r/428040 (https://phabricator.wikimedia.org/T179190) [17:39:15] (03CR) 10Krinkle: "Command 'npm run selenium-test' now exists in REL1_29..REL1_31 and master. Updated patch to only change that part of it." [integration/quibble] - 10https://gerrit.wikimedia.org/r/428040 (https://phabricator.wikimedia.org/T179190) (owner: 10Krinkle) [17:39:27] hashar: done ^ :) [17:41:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10LDAP-Access-Requests, and 4 others: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811#4150831 (10Smalyshev) We already have the user in LDAP (wdqsguibuilder) though for now it uses my em... [17:46:26] 10MediaWiki-Codesniffer: PHPCS should not complain about @covers and @dataProvider being used in traits - https://phabricator.wikimedia.org/T192384#4150843 (10daniel) @thiemowmde ah, with "local" you mean per-repo, not local to my system. But that would also disable the sniffer for classes. That's not what I in... [18:32:58] 10Release-Engineering-Team (Kanban), 10MediaWiki-SWAT-deployments, 10User-zeljkofilipin: Proposal: Effective immediately, disallow multi-sync patch deployment - https://phabricator.wikimedia.org/T187761#4150975 (10greg) Wordsmithing appreciated: https://wikitech.wikimedia.org/w/index.php?title=SWAT_deploys&a... [19:09:27] !log replacing deployment-kafka0[45] with deployment-kafka-main-[12] [19:09:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:26:46] hm, anyone here spawned a new instance in deployment-prep recently? [19:26:57] i'm getting certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster02.deployment-prep.eqiad.wmflabs [19:28:50] andrewbogott: maybe got some tips? [19:28:51] :) [19:29:47] ottomata: I don't know - if possibly I'd try to build a new VM with the normal puppetmaster for the first run and then move it over afterwards [19:30:00] with the normal puppetmaster? I haven't moved anytihng over [19:30:14] i think i see a first puppet run when the node booted [19:30:20] but i haven't changed anything puppetmaster related [19:30:33] i think deployment-prep maybe uses that puppetmaster for al nodes? [19:30:42] thanks anyway andrewbogott [19:30:51] who knows about deployment-prep puppetmaster? [19:30:58] thcipriani: ? [19:31:36] I think it’s because that project uses a puppet master [19:31:46] andrewbogott: ottomata ^^ [19:32:06] right... [19:32:16] So you have to do rm -rf /var/lib/puppet/ssl [19:32:19] On the instance [19:32:21] on the client again? [19:32:25] Then puppet agent -tv [19:32:27] huh. righht, new pupeptmasetr [19:32:29] then possibly sign [19:32:30] ok [19:32:38] And then you have to sign it on the puppet master [19:32:47] righiiiit, ok [19:32:50] the whole project uses this [19:32:56] Yep [19:32:58] dont' remember having to do this before but ya makes sense [19:33:38] puppet cert list should show what needs to be signed [19:33:53] Then do puppet cert sign [19:35:24] thanks paladox that did it [19:35:44] shoulda thought of that, but forgot (or didn't know) that there was extra work to do in deployment-prep because the whole project applies new puppetmaster by default [19:36:42] Your welcome :) [19:52:34] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston (Reporting): Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4151315 (10JAufrecht) 05Resolved>03Open The dump hasn't been updated since April 1, 2018. Perhaps it wasn't automated? ```... [19:59:37] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston (Reporting): Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4151350 (10mmodell) Strange. When I look on phab1001, the dump file appears to be updated today: ``` twentyafterfour@phab1001:/s... [19:59:48] just out of curiosity, when does enwiki go to wmf.30? [20:28:43] Project beta-scap-eqiad build #205072: 04FAILURE in 4 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205072/ [20:29:34] !log deployed to BC: [mobileapps/deploy@5650605]: Update mobileapps to b011b2a [20:29:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:30:29] AndyRussG: I was just about to update all wikis to wmf.30 [20:32:45] scap failure on beta [20:32:55] looking, somewhat [20:38:20] Project beta-scap-eqiad build #205073: 04STILL FAILING in 4 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205073/ [20:43:22] Project beta-scap-eqiad build #205074: 04STILL FAILING in 4 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205074/ [20:48:08] Reedy: is it safe to delete data under /temp? [20:48:17] What's /temp? [20:48:20] Project beta-scap-eqiad build #205075: 04STILL FAILING in 4 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205075/ [20:49:05] that ^ [20:49:19] T192849 [20:49:19] T192849: beta-scap-equiad failure: Error writing to CDB file - https://phabricator.wikimedia.org/T192849 [20:49:33] WException from line 132 of /srv/mediawiki-staging/php-master/includes/cache/localisation/LCStoreCDB.php: Error writing to CDB file "/tmp/scap_l10n_227763988/l10n_cache-pms.cdb.tmp.526492264". [20:50:10] reedy@deployment-tin:~$ cd /temp [20:50:10] -bash: cd: /temp: No such file or directory [20:53:32] 20:38:20 scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="dewiktionary" --outdir="/tmp/scap_l10n_227763988" --threads=6 --quiet' returned non-zero exit status 255 (duration: 04m 24s) [20:57:39] Project beta-scap-eqiad build #205076: 04STILL FAILING in 4 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205076/ [20:58:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: beta-scap-equiad failure: Error writing to CDB file - https://phabricator.wikimedia.org/T192849#4151627 (10MarcoAurelio) [21:03:32] Project beta-scap-eqiad build #205077: 04STILL FAILING in 4 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205077/ [21:08:15] Project beta-scap-eqiad build #205078: 04STILL FAILING in 4 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205078/ [21:09:10] Project mwext-phpunit-coverage-publish build #3698: 04FAILURE in 3.6 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3698/ [21:09:43] Project mwext-phpunit-coverage-publish build #3699: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3699/ [21:10:39] /temp or /tmp? [21:12:28] apergos: tmp [21:13:26] Project mwext-phpunit-coverage-publish build #3700: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3700/ [21:13:41] it is this wmf-insecte thing [21:13:46] idk what caused it [21:16:41] Project beta-scap-eqiad build #205079: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205079/ [21:17:02] Yippee, build fixed! [21:17:03] Project mwext-phpunit-coverage-publish build #3701: 09FIXED in 1 min 52 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3701/ [21:20:02] !log cleaned up 1.5G of space on deployment-tin:/tmp hopefully fixes beta-scap-eqiad [21:20:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:29] I was looking for that folder and couldn't get to it [21:21:20] Project mwext-phpunit-coverage-publish build #3703: 04FAILURE in 3.8 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3703/ [21:23:56] Yippee, build fixed! [21:23:56] Project mwext-phpunit-coverage-publish build #3704: 09FIXED in 2 min 16 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/3704/ [21:24:40] sweet [21:26:05] beta-scap-equiad is running now as well, looking into the console output [21:26:51] nice [21:27:39] thcipriani: for the record, it's not like 'sudo rm -Rf temp/*' right? [21:28:02] sorry, tmp/ [21:28:54] Hauskatze: I was doing some work on https://phabricator.wikimedia.org/T191921 earlier and left a bunch of stuff in /tmp that I deleted [21:29:29] I don't think you'd want to delete everything in /tmp since lots of programs store working stuff there [21:29:46] yeah, that's what I though, then just specific files [21:30:59] yep, stuff that looks old and unimportant, sudo rm /tmp/whatever [21:32:06] well, I'm not fond of deleting stuff, given that there's not a recycle bin you can recover items from in case of a mistake :) [21:32:53] Yippee, build fixed! [21:32:54] Project beta-scap-eqiad build #205080: 09FIXED in 9 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/205080/ [21:33:57] 10Release-Engineering-Team, 10Jenkins, 10Test-Coverage: PHP test coverage Jenkins report should congratulate success rather than focusing on failure - https://phabricator.wikimedia.org/T192853#4151694 (10Jdlrobson) [21:34:08] \o/ [21:35:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: beta-scap-equiad failure: Error writing to CDB file - https://phabricator.wikimedia.org/T192849#4151695 (10MarcoAurelio) 05Open>03Resolved a:03thcipriani Resolved by Tyler by deleting some stuff on the tmp folder. Thanks. [21:36:52] yay [21:50:21] (03CR) 10Hashar: "Whaaa thanks for the thorough analysis. During the sprint I merely tried various combination until it worked, probably because qunit and w" [integration/quibble] - 10https://gerrit.wikimedia.org/r/428356 (owner: 10Krinkle) [21:51:03] (03CR) 10Hashar: "Thanks for the backports. Will test on Tuesday and talk about it with Zeljko. I might as well tweak the selenium.sh script while at it." [integration/quibble] - 10https://gerrit.wikimedia.org/r/428040 (https://phabricator.wikimedia.org/T179190) (owner: 10Krinkle) [21:59:10] 10Release-Engineering-Team, 10Jenkins, 10Test-Coverage: PHP test coverage Jenkins report should congratulate success rather than focusing on failure - https://phabricator.wikimedia.org/T192853#4151730 (10hashar) > It's really awesome we track php test coverage. Definitely! The messages are reported in Gerr... [21:59:30] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Test-Coverage: PHP test coverage Jenkins report should congratulate success rather than focusing on failure - https://phabricator.wikimedia.org/T192853#4151732 (10hashar) [22:16:09] 10Continuous-Integration-Config, 10Mobile-Content-Service, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Create a CI task for MCS periodic tests - https://phabricator.wikimedia.org/T177896#4151760 (10Mholloway) It occurs to me that this could run on a separate Cloud VPS instance and n... [22:33:20] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston (Reporting): Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4151796 (10Dzahn) @20after4 Yes, deleting it and letting puppet recreate it is what i would do. Then we will see if it recreates,... [22:34:02] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Test-Coverage: PHP test coverage Jenkins report should congratulate success rather than focusing on failure - https://phabricator.wikimedia.org/T192853#4151798 (10Jdlrobson) @hashar Can those messages accept HTML/emojis/wikitext? I think droppin... [22:40:36] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston (Reporting): Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4151810 (10Dzahn) Deleted the one that did not log to /dev/null. Ran puppet; it did not recreate that. Looks to me like somebod... [22:41:05] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston (Reporting): Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#3997846 (10Paladox) According to T188646 they moved dumps.wikimedia.org from dataset1001 to labstore1006/7 [23:06:42] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Phlogiston (Reporting): Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4151910 (10Dzahn) Paladox is right. The location of dumps.wikimedia.org changed in the linked ticket above.... [23:08:52] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Phlogiston (Reporting): Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4151917 (10Dzahn) I manually ran the rsync command to fix it and upload to the right server. ``` root@pha... [23:20:47] Something is broken in Zuul/Jenkins and it's preventing me from merging a config patch [23:20:55] I tried +2ing https://gerrit.wikimedia.org/r/#/c/428508/ three times [23:21:18] 23:08:41 stderr: error: failed to write new configuration file .git/config.lock [23:23:16] thcipriani: there's enough free space on the slave that ran on ^ [23:23:37] thcipriani: anthing you're doing on contint1001 possibly effect that [23:24:11] not doing anything on contint1001 currently. There's space on that slave. [23:24:28] (yeah, second was a question, first was a statement, sorry) [23:26:23] speaking of free space, I just had to nuke the workspace on jessie-1003 [23:26:34] Any plans to bump up the disk allotment on those? [23:26:44] ejegg: did you log that? [23:27:01] because that's the host in question [23:27:02] greg-g: no, is there a log bot in this channel? [23:27:08] yes [23:27:14] and it goes to our SAL [23:27:21] when did you do it? [23:27:41] !log cleared out /srv/jenkins-workspace/workspace on jenkins-1003 [23:27:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:27:47] greg-g: about 5 minutes ago [23:27:50] RoanKattouw: that might have been it ^ try again? [23:28:26] the test dirs for android apps alone take like 5 gigs [23:28:27] Trying [23:28:38] and the whole disk seems to be <20 [23:29:07] I'll log future disk cleanups here, thanks for the tip greg-g [23:29:15] thanks for helping :) [23:30:22] RoanKattouw: thank ejegg :) [23:30:26] (it's fixed) [23:30:36] Aha! Yay thanks [23:30:40] :) yw [23:30:48] now, why has this (full disks) been happening more lately? [23:31:07] I say that based on my perception only, no data (another issue) [23:33:59] probably because we're back to all he same old problems of permanent jenkins agents rather than ephemeral instances [23:34:07] i.e., nodepool [23:34:20] right... [23:39:09] the main tracking task for this issue it seems: https://phabricator.wikimedia.org/T179963 with an umerged patch to use shallow clones for the lint jobs: https://gerrit.wikimedia.org/r/#/c/405722/ [23:45:38] 10Continuous-Integration-Config: Error fetching remote repo 'origin' - https://phabricator.wikimedia.org/T192860#4151999 (10Huji) [23:52:15] 10Continuous-Integration-Config: Error fetching remote repo 'origin' - https://phabricator.wikimedia.org/T192860#4151999 (10greg) This was happening a while ago until ejegg cleaned up some space on that specific job runner (jessie-1003). This should be resolved now. [23:52:49] 10Continuous-Integration-Config: Error fetching remote repo 'origin' - https://phabricator.wikimedia.org/T192860#4152017 (10greg) [23:52:56] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: mwgate-php55lint workspaces are getting huge - https://phabricator.wikimedia.org/T179963#4152020 (10greg) [23:56:49] ejegg: you don't happen to have a dh output from before you deleted, do you? For the whole workspace dir? [23:57:05] greg-g: shoot, no, sorry [23:57:28] no worries, I was just updating the task and I wanted to add more color like what you said about the apps workspaces [23:57:45] well, with proof :) [23:57:51] yeah, that observation was from a couple weeks ago [23:57:57] ah [23:58:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: mwgate-php55lint workspaces are getting huge - https://phabricator.wikimedia.org/T179963#4152022 (10greg) >>! In T179963#3917485, @gerritbot wrote: > Change 405722 had a related patch set uploaded (by Hashar; ow...