[01:10:00] Project beta-scap-eqiad build #309745: 04FAILURE in 5 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/309745/ [01:11:39] 02:09:53 01:09:53 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-deploy01.deployment-prep.eqiad.wmflabs::common/wikiversions*.{json,php} /srv/mediawiki on deployment-parsoid11.deployment-prep.eqiad.wmflabs returned [255]: Authentication failed. [01:19:28] Project beta-scap-eqiad build #309746: 04STILL FAILING in 4 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/309746/ [01:28:05] Yippee, build fixed! [01:28:05] Project beta-scap-eqiad build #309747: 09FIXED in 3 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/309747/ [01:36:31] (03CR) 10DannyS712: [C: 03+1] "Support this. Will +2 in a few days after time for any objections" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/615523 (owner: 10Esanders) [06:10:37] maintenance-disconnect-full-disks build 196609 integration-agent-docker-1011 (/: 18%, /srv: 98%, /var/lib/docker: 47%): OFFLINE due to disk space [06:15:51] maintenance-disconnect-full-disks build 196610 integration-agent-docker-1011 (/: 18%, /srv: 95%, /var/lib/docker: 60%): still OFFLINE due to disk space [06:20:28] maintenance-disconnect-full-disks build 196611 integration-agent-docker-1011 (/: 18%, /srv: 58%, /var/lib/docker: 51%): RECOVERY disk space OK [06:21:27] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [06:31:28] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [06:54:04] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T257969 (10Tacsipacsi) [07:03:36] good morning! anyone with logstash access that could get the stack trace for https://phabricator.wikimedia.org/T258666? [07:05:58] maintenance-disconnect-full-disks build 196620 integration-agent-docker-1011 (/: 17%, /srv: 100%, /var/lib/docker: 54%): OFFLINE due to disk space [07:07:26] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [07:10:39] maintenance-disconnect-full-disks build 196621 integration-agent-docker-1011 (/: 17%, /srv: 44%, /var/lib/docker: 47%): RECOVERY disk space OK [07:17:29] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [07:30:30] maintenance-disconnect-full-disks build 196625 integration-agent-docker-1011 (/: 17%, /srv: 98%, /var/lib/docker: 58%): OFFLINE due to disk space [07:35:30] maintenance-disconnect-full-disks build 196626 integration-agent-docker-1001 (/: 21%, /srv: 96%, /var/lib/docker: 53%): OFFLINE due to disk space [07:35:30] maintenance-disconnect-full-disks build 196626 integration-agent-docker-1011 (/: 17%, /srv: 72%, /var/lib/docker: 62%): RECOVERY disk space OK [07:40:23] PROBLEM - Free space - all mounts on integration-agent-docker-1001 is CRITICAL: CRITICAL: integration.integration-agent-docker-1001.diskspace._srv.byte_percentfree (<33.33%) [07:45:32] maintenance-disconnect-full-disks build 196628 integration-agent-docker-1001 (/: 21%, /srv: 44%, /var/lib/docker: 53%): RECOVERY disk space OK [07:55:26] RECOVERY - Free space - all mounts on integration-agent-docker-1001 is OK: OK: All targets OK [08:00:24] PROBLEM - Free space - all mounts on integration-agent-docker-1009 is CRITICAL: CRITICAL: integration.integration-agent-docker-1009.diskspace._srv.byte_percentfree (<11.11%) [08:05:23] RECOVERY - Free space - all mounts on integration-agent-docker-1009 is OK: OK: All targets OK [08:54:12] 10Gerrit: Gitiles project landing pages should have an anonymous clone URL - https://phabricator.wikimedia.org/T206049 (10ema) >>! In T206049#6328111, @QChris wrote: > However, the URL can currently not be configured. It's hardwired to SSH, if SSH is available. > > Not sure if anyone is working on it or plannin... [08:54:24] 10Gerrit, 10Release-Engineering-Team: Gitiles project landing pages should have an anonymous clone URL - https://phabricator.wikimedia.org/T206049 (10ema) [09:31:26] PROBLEM - Free space - all mounts on integration-agent-docker-1009 is CRITICAL: CRITICAL: integration.integration-agent-docker-1009.diskspace._srv.byte_percentfree (<11.11%) [09:36:26] RECOVERY - Free space - all mounts on integration-agent-docker-1009 is OK: OK: All targets OK [09:53:14] 10Continuous-Integration-Config, 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10User-Vidhi-Mody, 10User-zeljkofilipin: CI not running for phabricator/deployment - https://phabricator.wikimedia.org/T258531 (10zeljkofilipin) [10:07:26] PROBLEM - Free space - all mounts on integration-agent-docker-1009 is CRITICAL: CRITICAL: integration.integration-agent-docker-1009.diskspace._srv.byte_percentfree (<11.11%) [10:12:25] RECOVERY - Free space - all mounts on integration-agent-docker-1009 is OK: OK: All targets OK [10:25:26] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<22.22%) [10:35:25] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [10:45:35] maintenance-disconnect-full-disks build 196664 integration-agent-docker-1009 (/: 17%, /srv: 95%, /var/lib/docker: 66%): OFFLINE due to disk space [10:50:34] maintenance-disconnect-full-disks build 196665 integration-agent-docker-1009 (/: 17%, /srv: 85%, /var/lib/docker: 67%): RECOVERY disk space OK [11:51:23] PROBLEM - Free space - all mounts on integration-agent-docker-1001 is CRITICAL: CRITICAL: integration.integration-agent-docker-1001.diskspace._srv.byte_percentfree (<11.11%) [11:52:38] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T257969 (10LarsWirzenius) Reported {T258696} about 16000+ database errors, but those hap... [12:01:24] RECOVERY - Free space - all mounts on integration-agent-docker-1001 is OK: OK: All targets OK [12:51:40] PROBLEM - Free space - all mounts on integration-castor03 is CRITICAL: CRITICAL: integration.integration-castor03.diskspace._srv.byte_percentfree (<11.11%) [13:46:28] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [13:50:29] maintenance-disconnect-full-disks build 196701 integration-agent-docker-1011 (/: 18%, /srv: 96%, /var/lib/docker: 68%): OFFLINE due to disk space [13:52:51] 10Release-Engineering-Team, 10ApiFeatureUsage, 10Commons, 10Pywikibot, and 2 others: Persistant error 500 getting category members - https://phabricator.wikimedia.org/T255981 (10CDanis) This issue has been ongoing for a while and likely merits some CPT attention. From #wikimedia-operations: `lang=irc 09:3... [13:56:23] 10Release-Engineering-Team, 10ApiFeatureUsage, 10Commons, 10Pywikibot, and 2 others: Persistant error 500 getting category members - https://phabricator.wikimedia.org/T255981 (10AMooney) p:05Triage→03Medium [14:00:29] maintenance-disconnect-full-disks build 196703 integration-agent-docker-1011 (/: 18%, /srv: 64%, /var/lib/docker: 58%): RECOVERY disk space OK [14:05:24] PROBLEM - Free space - all mounts on integration-agent-docker-1009 is CRITICAL: CRITICAL: integration.integration-agent-docker-1009.diskspace._srv.byte_percentfree (<11.11%) [14:06:28] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [14:10:32] maintenance-disconnect-full-disks build 196705 integration-agent-docker-1009 (/: 17%, /srv: 96%, /var/lib/docker: 57%): OFFLINE due to disk space [14:26:27] 10Phabricator: Indecipherable red icon in task subtitle - https://phabricator.wikimedia.org/T258712 (10Tgr) [14:26:46] 10Phabricator: Indecipherable red icon in task subtitle - https://phabricator.wikimedia.org/T258712 (10Tgr) (Seen on {T258650}) [14:28:48] 10Phabricator: Indecipherable red icon in task subtitle - https://phabricator.wikimedia.org/T258712 (10Majavah) [14:28:51] 10Phabricator, 10Accessibility: Production Error task subtype symbol appearing on tasks, indicating something urgent, but is non-interactive - https://phabricator.wikimedia.org/T256995 (10Majavah) [14:30:39] maintenance-disconnect-full-disks build 196709 integration-agent-docker-1009 (/: 17%, /srv: 84%, /var/lib/docker: 47%): RECOVERY disk space OK [14:30:40] maintenance-disconnect-full-disks build 196709 integration-agent-docker-1011 (/: 18%, /srv: 97%, /var/lib/docker: 50%): OFFLINE due to disk space [14:32:27] PROBLEM - Free space - all mounts on integration-agent-docker-1011 is CRITICAL: CRITICAL: integration.integration-agent-docker-1011.diskspace._srv.byte_percentfree (<11.11%) [14:35:50] maintenance-disconnect-full-disks build 196710 integration-agent-docker-1009 (/: 17%, /srv: 100%, /var/lib/docker: 47%): OFFLINE due to disk space [14:35:50] maintenance-disconnect-full-disks build 196710 integration-agent-docker-1011 (/: 18%, /srv: 57%, /var/lib/docker: 51%): RECOVERY disk space OK [14:40:33] CI seems to be running out of space a lot today :/ [14:42:23] PROBLEM - Free space - all mounts on integration-agent-docker-1001 is CRITICAL: CRITICAL: integration.integration-agent-docker-1001.diskspace._srv.byte_percentfree (<11.11%) [14:43:17] There are only 4 unused executors across the integration-agent-docker-* nodes. [14:43:25] looks like something big was triggered. [14:45:31] 10Release-Engineering-Team, 10ApiFeatureUsage, 10Commons, 10Pywikibot, and 2 others: Persistant error 500 getting category members - https://phabricator.wikimedia.org/T255981 (10Xover) For the FileImporter issue mentioned on IRC see https://www.mediawiki.org/wiki/Topic:Vqmtib4ho61z4bji for two files that f... [14:45:47] maintenance-disconnect-full-disks build 196712 integration-agent-docker-1009 (/: 17%, /srv: 84%, /var/lib/docker: 47%): RECOVERY disk space OK [14:47:27] RECOVERY - Free space - all mounts on integration-agent-docker-1011 is OK: OK: All targets OK [14:50:44] maintenance-disconnect-full-disks build 196713 integration-agent-docker-1009 (/: 17%, /srv: 95%, /var/lib/docker: 53%): OFFLINE due to disk space [14:51:38] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata-Campsite, 10MW-1.31-release: Score fails in 1.31 due to WikibaseCirrusSearch - https://phabricator.wikimedia.org/T258715 (10Reedy) Basically, there's no REL1_31 branch, so it falls back to master... :( And master of the extensions extension.json has.... [14:52:06] 10MediaWiki-Releasing, 10Release: Filenames truncated in 1.34.2 tarball release (The Unarchiver bug!) - https://phabricator.wikimedia.org/T258716 (10Andru) [14:52:26] RECOVERY - Free space - all mounts on integration-agent-docker-1001 is OK: OK: All targets OK [14:52:57] 10MediaWiki-Releasing, 10Documentation, 10Release: Filenames truncated in 1.34.2 tarball release (The Unarchiver bug!) - https://phabricator.wikimedia.org/T258716 (10Reedy) [14:54:02] 10MediaWiki-Releasing, 10Documentation, 10Release: Filenames truncated in 1.34.2 tarball release (The Unarchiver bug!) - https://phabricator.wikimedia.org/T258716 (10Reedy) >While this isn't a MediaWiki bug, it may be worthwhile documenting this on the release page for MacOS users, alongside the tip for Wind... [14:55:34] maintenance-disconnect-full-disks build 196714 integration-agent-docker-1009 (/: 17%, /srv: 8%, /var/lib/docker: 44%): RECOVERY disk space OK [14:56:34] James_F: We don't have a standardised way to do dynamic injection of dependancies do we? [14:56:40] like, only include them in X MW branch [15:00:22] RECOVERY - Free space - all mounts on integration-agent-docker-1009 is OK: OK: All targets OK [15:03:49] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata-Campsite, 10MW-1.31-release: Score fails in 1.31 due to WikibaseCirrusSearch - https://phabricator.wikimedia.org/T258715 (10EBernhardson) WikibaseCirrusSearch didn't exist when 1.31 was released, it shouldn't be installed there. WBCS was first deploy... [15:09:29] 10Continuous-Integration-Config, 10Wikidata, 10MW-1.31-release: Score fails in 1.31 due to WikibaseCirrusSearch - https://phabricator.wikimedia.org/T258715 (10Reedy) p:05Triage→03High >>! In T258715#6329887, @EBernhardson wrote: > WikibaseCirrusSearch didn't exist when 1.31 was released, it shouldn't be... [15:13:01] (03PS1) 10Reedy: Remove WikibaseCirrusSearch from REL1_31 extension dependancies [integration/config] - 10https://gerrit.wikimedia.org/r/615763 (https://phabricator.wikimedia.org/T258715) [15:14:12] (03CR) 10Reedy: [C: 03+2] Remove WikibaseCirrusSearch from REL1_31 extension dependancies [integration/config] - 10https://gerrit.wikimedia.org/r/615763 (https://phabricator.wikimedia.org/T258715) (owner: 10Reedy) [15:17:38] (03Merged) 10jenkins-bot: Remove WikibaseCirrusSearch from REL1_31 extension dependancies [integration/config] - 10https://gerrit.wikimedia.org/r/615763 (https://phabricator.wikimedia.org/T258715) (owner: 10Reedy) [15:18:10] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/615763 [15:18:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:28:44] 10Release-Engineering-Team, 10ApiFeatureUsage, 10Commons, 10Pywikibot, and 2 others: Persistant error 500 getting category members - https://phabricator.wikimedia.org/T255981 (10Xover) The FileImporter issue is apparently T258666 and not obviously related. [15:36:06] 10Continuous-Integration-Config, 10Wikidata, 10MW-1.31-release, 10Patch-For-Review: Score fails in 1.31 due to WikibaseCirrusSearch - https://phabricator.wikimedia.org/T258715 (10Reedy) 05Open→03Resolved a:03Reedy There's other failures now. But this one is cleared [15:43:02] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10zeljkofilipin) [16:18:01] Is phab search broken? [16:18:08] Any time I use colons, it seems to get upset [16:19:49] Reedy: elaborate, please? [16:19:59] example query; URL? [16:19:59] search for "TimeParsingTest::testSeconds2NptFormat" [16:20:02] Unknown search function "timeparsingtest". Supported functions are: all, title, body, core, comment. [16:20:05] https://phabricator.wikimedia.org/search/query/xwBb_yz.DYfI/#R [16:20:49] Reedy: use quotation marks around the string [16:21:05] Is that a recent change? [16:21:07] (plus don't use the horrible global search but https://phabricator.wikimedia.org/maniphest/query/advanced/ ) [16:21:16] a few months ago IIRC [16:21:27] error message has been already improved :P [16:32:28] 10MediaWiki-Releasing, 10Documentation, 10Release: Filenames truncated in 1.34.2 tarball release (The Unarchiver bug!) - https://phabricator.wikimedia.org/T258716 (10Andru) It doesn't appear to be the same issue. I should have been more explicit in my suggestion - I'm referring to this template: https://ww... [16:35:33] 10MediaWiki-Releasing, 10Documentation, 10Release: Filenames truncated in 1.34.2 tarball release (The Unarchiver bug!) - https://phabricator.wikimedia.org/T258716 (10Andru) [16:39:22] (03CR) 10Jdlrobson: [C: 03+1] "thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/615503 (https://phabricator.wikimedia.org/T258528) (owner: 10Edward Tadros) [16:45:15] 10MediaWiki-Releasing, 10Documentation, 10Release: Filenames truncated in 1.34.2 tarball release (The Unarchiver bug!) - https://phabricator.wikimedia.org/T258716 (10Reedy) I think I added a "not" in there in my head The docs say to use 7-zip on windows, but due to {T257102], they shouldn't... [17:43:47] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 715 bytes in 0.087 second response time [17:53:40] 10Project-Admins: create tag and a group for wikinews (any language) for bug tracking at phabricator - https://phabricator.wikimedia.org/T256735 (10Aklapper) 05Stalled→03Declined Unfortunately closing this Phabricator task as no further information has been provided. @Gryllida: After you have provided the i... [17:54:18] 10Project-Admins: create tag and a group for wikinews (any language) for bug tracking at phabricator - https://phabricator.wikimedia.org/T256735 (10Aklapper) For the records, I'd like to avoid the impression that tasks would be looked at and worked on *if* that's not necessarily the case. Also see previous discu... [18:09:55] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests: Increase wgAbuseFilterEmergencyDisable on zh Beta Cluster - https://phabricator.wikimedia.org/T230305 (10Urbanecm) Hmm... I see there is no comment on this for a very long time. @Xiplus: Is this still needed? @Daimona: I'm not sure who should be the d... [18:24:18] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests: Increase wgAbuseFilterEmergencyDisable on zh Beta Cluster - https://phabricator.wikimedia.org/T230305 (10Daimona) T210151 is related, this setting is a mess... Increasing the limit for this zhbeta only is fine, if it's urgent. [19:19:01] 10Release-Engineering-Team, 10MediaWiki-extensions-OAuthRateLimiter, 10Platform Team Initiatives (API Gateway), 10Platform Team Workboards (Green): Deploy OAuthRateLimiter extension to Wikimedia Production - https://phabricator.wikimedia.org/T258423 (10Pchelolo) [19:48:23] 10VPS-project-codesearch, 10Discovery, 10Discovery-Search, 10Documentation: Consider adding "codesearch" and search for doc.wikimedia.org to mediawiki.org - https://phabricator.wikimedia.org/T240209 (10CBogen) [19:48:54] 10VPS-project-codesearch, 10Discovery, 10Discovery-Search, 10Documentation: Consider adding "codesearch" and search for doc.wikimedia.org to mediawiki.org - https://phabricator.wikimedia.org/T240209 (10CBogen) Since this is better solved by a UI change, the Search Platform team doesn't have the resources t... [20:00:20] 10Release-Engineering-Team, 10MediaWiki-extensions-FlaggedRevs, 10Regression: FlaggedRevs: Automatic user promotion stopped working on some wikis on June 24, 2019 - https://phabricator.wikimedia.org/T237191 (10Wargo) Why these config variables can't be moved outside of wgExtensionFunctions, and, for example,... [20:04:45] (03PS1) 10Ahmon Dancy: New maintenance-sample-workspace-sizes job [integration/config] - 10https://gerrit.wikimedia.org/r/615830 (https://phabricator.wikimedia.org/T258626) [20:05:37] (03CR) 10jerkins-bot: [V: 04-1] New maintenance-sample-workspace-sizes job [integration/config] - 10https://gerrit.wikimedia.org/r/615830 (https://phabricator.wikimedia.org/T258626) (owner: 10Ahmon Dancy) [20:10:00] (03PS2) 10Ahmon Dancy: New maintenance-sample-workspace-sizes job [integration/config] - 10https://gerrit.wikimedia.org/r/615830 (https://phabricator.wikimedia.org/T258626) [20:19:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 716 bytes in 0.551 second response time [20:19:37] PROBLEM - Host deployment-maps05 is DOWN: CRITICAL - Host Unreachable (172.16.4.168) [20:33:09] 10Phabricator, 10Research-Backlog: Fix Research component tags to make them non-mutually-exclusive - https://phabricator.wikimedia.org/T238258 (10Aklapper) @leila: Is anything else needed in this task? (Sorry, I lost overview a bit I admit.) Thanks in advance! [20:50:16] 10Phabricator, 10Research-Backlog: Fix Research component tags to make them non-mutually-exclusive - https://phabricator.wikimedia.org/T238258 (10leila) @Aklapper we're good. I'll resolve. [20:50:31] 10Phabricator, 10Research-Backlog: Fix Research component tags to make them non-mutually-exclusive - https://phabricator.wikimedia.org/T238258 (10leila) 05Open→03Resolved [20:57:01] Andre just left! [21:00:43] what happening? [21:02:46] Majavah: was gonna ask him something [21:03:06] * RhinosF1 worked it out [21:03:19] Thank god commons has a large deletion backlog [21:04:53] well it took couple of weeks to get a batch of copvios gone [21:05:51] 2 months 8 days I've had a file waiting for deletion [21:24:29] (03CR) 10Jeena Huneidi: "What do you think about adding a test for multi-level copying? In reader_test.go" [blubber] - 10https://gerrit.wikimedia.org/r/613701 (https://phabricator.wikimedia.org/T254629) (owner: 10Ahmon Dancy) [21:29:49] (03CR) 10Jeena Huneidi: "> Patch Set 3:" [blubber] - 10https://gerrit.wikimedia.org/r/613701 (https://phabricator.wikimedia.org/T254629) (owner: 10Ahmon Dancy) [21:45:58] 10MediaWiki-Codesniffer, 10phan: Add rule against use unsafe use of Exception::getTraceAsString() - https://phabricator.wikimedia.org/T258755 (10Krinkle) [23:15:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 716 bytes in 0.130 second response time [23:15:36] PROBLEM - Host deployment-maps05 is DOWN: CRITICAL - Host Unreachable (172.16.4.168)