[00:02:12] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: The Fresnel job output (linked in Gerrit) should not start with unrelated setup steps - https://phabricator.wikimedia.org/T220561 (10Krinkle) 05Open→03Resolved Now: {F28599709 height=400} [00:20:10] maintenance-disconnect-full-disks build 62340 integration-slave-docker-1043: OFFLINE due to disk space [00:45:11] maintenance-disconnect-full-disks build 62345 integration-slave-docker-1043: OFFLINE due to disk space [01:10:12] maintenance-disconnect-full-disks build 62350 integration-slave-docker-1043: OFFLINE due to disk space [01:26:14] 10Project-Admins: Create a tag for new Community Tech project 'Night Mode - https://phabricator.wikimedia.org/T220571 (10jmatazzoni) [01:35:13] maintenance-disconnect-full-disks build 62355 integration-slave-docker-1043: OFFLINE due to disk space [02:00:13] maintenance-disconnect-full-disks build 62360 integration-slave-docker-1043: OFFLINE due to disk space [02:25:13] maintenance-disconnect-full-disks build 62365 integration-slave-docker-1043: OFFLINE due to disk space [02:50:12] maintenance-disconnect-full-disks build 62370 integration-slave-docker-1043: OFFLINE due to disk space [03:15:12] maintenance-disconnect-full-disks build 62375 integration-slave-docker-1043: OFFLINE due to disk space [03:40:11] maintenance-disconnect-full-disks build 62380 integration-slave-docker-1043: OFFLINE due to disk space [04:05:11] maintenance-disconnect-full-disks build 62385 integration-slave-docker-1043: OFFLINE due to disk space [04:30:11] maintenance-disconnect-full-disks build 62390 integration-slave-docker-1043: OFFLINE due to disk space [04:55:12] maintenance-disconnect-full-disks build 62395 integration-slave-docker-1043: OFFLINE due to disk space [05:02:14] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10Marostegui) p:05High→03Unbreak! mwdebug2001 and mwdebug2002 are now full: ` root@mwdebug2002:~# df -hT / Filesystem Type Size Used... [05:02:21] 10Release-Engineering-Team (Backlog), 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10Marostegui) p:05Normal→03Unbreak! Both mwdebug2001 and 2002 are now full: T218783#5099940 [05:19:53] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10greg) Since these are vms, can their disk be expanded easily? I ask because they're the oddballs in the mw fleet :) I know Tyler wants to fi... [05:20:10] maintenance-disconnect-full-disks build 62400 integration-slave-docker-1043: OFFLINE due to disk space [05:22:01] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10Joe) To be precise, the last version of the train did not deploy correctly to any of the debug servers. Deployments should be considered bl... [05:23:07] 10Release-Engineering-Team (Backlog), 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10greg) Since these are vms can a quick fix be to expand their disk? Since they're the oddballs of the mw fleet in that way. I know Tyler wants to fix the scap clean issu... [05:23:18] I'll stop commenting, I'm tired.... [05:23:30] 10Release-Engineering-Team (Backlog), 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10Joe) Also the last mediawiki train didn't deploy correctly to those servers and will never be able to unless we remove old versions. [05:38:54] 10Release-Engineering-Team (Backlog), 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10Joe) >>! In T219989#5099962, @greg wrote: > Since these are vms can a quick fix be to expand their disk? Since they're the oddballs of the mw fleet in that way. I know... [05:45:12] maintenance-disconnect-full-disks build 62405 integration-slave-docker-1043: OFFLINE due to disk space [05:52:56] 10Release-Engineering-Team (Backlog), 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10Joe) p:05Unbreak!→03High [05:53:38] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10Joe) Also please note that with the next train, the eqiad servers would fill their disk up as well. [05:55:07] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10Joe) >>! In T218783#5099959, @greg wrote: > Since these are vms, can their disk be expanded easily? I ask because they're the oddballs in th... [06:10:12] maintenance-disconnect-full-disks build 62410 integration-slave-docker-1043: OFFLINE due to disk space [06:18:46] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<50.00%) [06:35:11] maintenance-disconnect-full-disks build 62415 integration-slave-docker-1043: OFFLINE due to disk space [06:53:47] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:55:39] (03CR) 10Hashar: "I have not really carefully post reviewed this change, but by reading the commit message: THANK YOU TIMO :]" [integration/config] - 10https://gerrit.wikimedia.org/r/496688 (owner: 10Krinkle) [07:00:11] maintenance-disconnect-full-disks build 62420 integration-slave-docker-1043: OFFLINE due to disk space [07:25:26] maintenance-disconnect-full-disks build 62425 integration-slave-docker-1043: OFFLINE due to disk space [07:50:32] maintenance-disconnect-full-disks build 62430 integration-slave-docker-1043: OFFLINE due to disk space [08:14:09] <_joe_> uhmm hashar is not here right now [08:15:12] maintenance-disconnect-full-disks build 62435 integration-slave-docker-1043: OFFLINE due to disk space [08:19:46] 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2019: "Move Tasks to Column..." opens dialogue for moving to project - https://phabricator.wikimedia.org/T220585 (10Sebastian_Berlin-WMSE) [08:25:41] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: The Fresnel job output (linked in Gerrit) should not start with unrelated setup steps - https://phabricator.wikimedia.org/T220561 (10hashar) Nice! I am also wondering whether we should have Quibble to output predetermined markers which would ma... [08:40:12] maintenance-disconnect-full-disks build 62440 integration-slave-docker-1043: OFFLINE due to disk space [09:01:02] 10Scap, 10Scoring-platform-team (Research): Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281 (10akosiaris) >>! In T188281#5099397, @Harej wrote: > @akosiaris Should we still pursue this? With the move to a different deployment tool in the pipeline it's gonna... [09:05:11] maintenance-disconnect-full-disks build 62445 integration-slave-docker-1043: OFFLINE due to disk space [09:30:11] maintenance-disconnect-full-disks build 62450 integration-slave-docker-1043: OFFLINE due to disk space [09:55:11] maintenance-disconnect-full-disks build 62455 integration-slave-docker-1043: OFFLINE due to disk space [10:20:12] maintenance-disconnect-full-disks build 62460 integration-slave-docker-1043: OFFLINE due to disk space [10:45:13] maintenance-disconnect-full-disks build 62465 integration-slave-docker-1043: OFFLINE due to disk space [11:10:11] maintenance-disconnect-full-disks build 62470 integration-slave-docker-1043: OFFLINE due to disk space [11:35:14] maintenance-disconnect-full-disks build 62475 integration-slave-docker-1043: OFFLINE due to disk space [11:44:42] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [12:00:12] maintenance-disconnect-full-disks build 62480 integration-slave-docker-1043: OFFLINE due to disk space [12:08:03] Is the suddenly long ci job times known? I just had wmf-quibble-core-vendor-mysql-hhvm-docker job take 11 m to finish amongst other mw-core ci taking around that? [12:22:44] Zppix: hashar is working on it [12:23:12] Zppix: 11 minutes seems about the average time? [12:23:28] hashar: no thats the highest time on that change [12:23:39] hashar: https://gerrit.wikimedia.org/r/486129 [12:23:42] there are some backlog in patch-performance and coverage pipelines, but they are low priority so [12:24:08] I am gonna upgrade jenkins anyway [12:27:14] hmm [12:27:21] so yeah [12:27:28] updating mediawiki/core repo takes like a full minute [12:27:32] which is ... not fast eough [12:27:46] * apergos cheers hashar on [12:28:37] so that is the zuul-merger throttling down ci :/ [12:30:04] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [12:38:49] * Krinkle misses wikibugs [12:40:19] whats wrong with wikibugs Krinkle ? [12:40:25] It's not here. [12:40:37] !log contint2001: stopped puppet and zuul-merger for debugging [12:40:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:42:50] it needs restarting [12:43:32] PROBLEM - zuul_merger_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [12:43:51] paladox: wikibugs? if so its in -cloud so it just may not have anything to put in here yet [12:45:06] ACKNOWLEDGEMENT - zuul_merger_service_running on contint2001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger amusso maintenance ongoing for zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [12:46:40] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502786 [12:46:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:47:07] Krinkle: i am pretty sure hashar was doing stuff with zuul? [12:47:41] 2019-04-10 12:38:49,366 DEBUG zuul.Repo: Updating repository /srv/zuul/git/mediawiki/core [12:47:41] 2019-04-10 12:39:38,241 DEBUG zuul.Repo: Checking out c51e4e24ef4ce5a1ef9933fd20266e75a758027f [12:47:58] so 50 seconds to do a `git fetch --tags` if I get it right [12:47:58] :/ [12:48:33] hashar: is there not a way to cache somethings to make it faster? [12:58:18] (03CR) 10Krinkle: [C: 03+2] dockerfiles: Make git-checkout quiet in Fresnel for MediaWiki jobs [integration/config] - 10https://gerrit.wikimedia.org/r/502790 (owner: 10Krinkle) [12:59:16] RECOVERY - zuul_merger_service_running on contint2001 is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [12:59:53] (03Merged) 10jenkins-bot: dockerfiles: Make git-checkout quiet in Fresnel for MediaWiki jobs [integration/config] - 10https://gerrit.wikimedia.org/r/502790 (owner: 10Krinkle) [13:00:23] !log Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/502790 [13:00:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:04:52] oh my [13:05:15] hashar: what did i break now? [13:05:50] Zppix: i don't know? [13:06:05] hashar: i was referring to your "oh my" [13:06:20] (03PS7) 10Krinkle: Use php72 instead of php70 for mw/sqlite job (4) [integration/config] - 10https://gerrit.wikimedia.org/r/499370 [13:08:32] (03CR) 10jerkins-bot: [V: 04-1] Use php72 instead of php70 for mw/sqlite job (4) [integration/config] - 10https://gerrit.wikimedia.org/r/499370 (owner: 10Krinkle) [13:12:12] (03PS8) 10Krinkle: Use php72 instead of php70 for mw/sqlite job (4) [integration/config] - 10https://gerrit.wikimedia.org/r/499370 [13:12:18] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) [13:13:50] (03CR) 10jerkins-bot: [V: 04-1] Use php72 instead of php70 for mw/sqlite job (4) [integration/config] - 10https://gerrit.wikimedia.org/r/499370 (owner: 10Krinkle) [13:16:09] 10Continuous-Integration-Infrastructure, 10Zuul, 10Tracking-Neverending, 10WorkType-Maintenance: Scale merge operations (tracking) - https://phabricator.wikimedia.org/T70480 (10hashar) [13:16:10] 10Continuous-Integration-Infrastructure, 10Zuul, 10Upstream: Zuul repositories have too many refs causing slow updates - https://phabricator.wikimedia.org/T70481 (10hashar) 05Open→03Resolved a:03hashar Something somehow got enhanced and it is way faster nowadays. Either due to gallium disk that was slo... [13:16:22] (03PS9) 10Krinkle: Use php72 instead of php70 for mw/sqlite job (4) [integration/config] - 10https://gerrit.wikimedia.org/r/499370 [13:16:35] 10Continuous-Integration-Infrastructure, 10Zuul, 10Upstream: Zuul repositories have too many refs causing slow updates - https://phabricator.wikimedia.org/T70481 (10hashar) Another slowness in zuul-merger is {T220606} [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:18:43] hashar: So, what are the quibble flavours again? Is this correct: wmf-quibble-* is extension gate, mediawiki-quibble-* is plain core+Vector, release-quibble is mw tarball stuff, and... regular quibble- is... (I don't know). [13:19:30] I'm also note sure what 'wmf-quibble-core' is. [13:22:47] (03CR) 10Krinkle: [C: 03+2] Use php72 instead of php70 for mw/sqlite job (4) [integration/config] - 10https://gerrit.wikimedia.org/r/499370 (owner: 10Krinkle) [13:24:27] (03Merged) 10jenkins-bot: Use php72 instead of php70 for mw/sqlite job (4) [integration/config] - 10https://gerrit.wikimedia.org/r/499370 (owner: 10Krinkle) [13:24:40] (03PS1) 10Krinkle: jjb: Update fresnel job to quibble-fresnel:0.0.31-2 [integration/config] - 10https://gerrit.wikimedia.org/r/502793 [13:25:31] (03CR) 10Krinkle: [C: 03+2] "Updated 1 Jenkins job(s)." [integration/config] - 10https://gerrit.wikimedia.org/r/502793 (owner: 10Krinkle) [13:28:22] (03Merged) 10jenkins-bot: jjb: Update fresnel job to quibble-fresnel:0.0.31-2 [integration/config] - 10https://gerrit.wikimedia.org/r/502793 (owner: 10Krinkle) [13:35:59] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) [13:37:24] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:42:56] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10thcipriani) >>! In T218783#5099959, @greg wrote: > Since these are vms, can their disk be expanded easily? I ask because they're the oddball... [13:52:39] <_joe_> hashar: can you point me to your (== releng's) scripts to use docker-pkg? [13:52:47] <_joe_> I'd like to deploy version 2.0 today [13:52:56] <_joe_> but I'll need to change the CLI invokations [14:06:45] <_joe_> also I noticed, we're running tests on HHVM and php 7.0 [14:06:49] <_joe_> why not 7.2? [14:07:27] not 7.2? 7.2 is actuallly in use [14:07:58] <_joe_> I'm looking at a core patch [14:08:14] <_joe_> and I see "php7.0" and "hhvm" in the tests [14:10:55] I guess php 7.0 as min supported... [14:11:03] Weren't we doing some 7.2 tests at some point? [14:11:08] I thought we were [14:11:23] gate-and-submit seems to have php70, php71 and php72 [14:11:23] and we have our own built php7.2 packages so we ought to be using those, right? [14:11:39] but the test pipeline is smaller [14:11:50] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/481501/ [14:11:53] <_joe_> ok but [14:11:57] mediawiki-quibble-vendor-mysql-php70-docker SUCCESS in 12m 13s [14:11:57] mediawiki-quibble-vendor-mysql-php71-docker SUCCESS in 7m 13s [14:11:57] mediawiki-quibble-vendor-mysql-php72-docker SUCCESS in 6m 47s [14:12:04] <_joe_> we're supporting php7.2 in prod [14:12:12] <_joe_> it's faster [14:12:25] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) Seems like `zuul-merger` does not prune stall references: `lines=10 zuul@contint2001:/srv/zuul/git/mediawiki/core$ git remote prune -n origin Pruning... [14:12:26] <_joe_> can't we support that instead on a patch-by-patch basis? [14:13:09] !log contint2001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606 [14:13:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:13:11] T220606: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 [14:13:15] <_joe_> I mean imagine I'm writing a patch and it's incompatible with php 7.2, I probably want to know? [14:13:57] we should be testing 7.3 etc too ;) [14:14:06] What do you mean by a patch by patch basis? [14:14:10] Force it to run the PHP 7.2 tests? [14:14:21] "check gate" or "check experimental"? [14:15:56] <_joe_> no I mean before I wait for someone to +2 it [14:16:06] <_joe_> but maybe I'm too focused on production [14:16:08] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) Down to just 10 seconds after running `git remote prune origin`. ` 2019-04-10 14:12:33,095 DEBUG zuul.Repo: Resetting repository /srv/zuul/git/mediaw... [14:16:17] !log contint1001: sudo -u zuul git -C /srv/zuul/git/mediawiki/core remote prune origin # T220606 [14:16:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:17:45] while people are in here chatting... what's the status of the train? I know it's stopped but I couldn't figure out what is blocking, even from looking at the blocking tickets [14:21:11] <_joe_> apergos: https://www.youtube.com/watch?v=X75CmqkjCyE [14:21:55] ty, there oughta be a soundtrack [14:27:10] https://www.youtube.com/watch?v=rLs_0h9_wXM [14:28:47] on the wrong track and heading for you [14:37:36] hey all, wanted to skip jenkins here [14:37:36] https://gerrit.wikimedia.org/r/#/c/cergen/+/502799/ [14:37:38] because it is wrong [14:37:44] but ic an't remove it as reviewer anymore... [14:43:38] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Milimetric) I would say so, @dr0ptp4kt, I'd maybe even go so far as to host a graph edit-a-thon to upgrade all v1.5 and... [14:44:33] <_joe_> oh interesting [14:44:47] <_joe_> aren't you the owner of that repo? [14:47:15] Nope, Gerrit-Managers is though (which ottomata is not in), though i can add a group that has him in. [14:47:28] yes plz! [14:48:10] ottomata which group? [14:49:00] paladox: hm, this is kinda of ops software stuff... [14:49:05] looking ata groups... [14:49:10] so ldap/ops? [14:49:39] hmm maybe opssoftware? [14:49:43] i can ad mysefl, there? [14:49:52] just did that. [14:50:01] oh, yea [14:50:02] i see [14:50:04] ldap/ops sounds good [14:50:12] ottomata done [14:50:15] danke [14:50:26] either one danke [14:50:50] paladox: while we're here... i can't create gerrit projects anymore [14:50:53] is ithat known? [14:51:04] that's because your no longer a admin [14:51:12] you need to be added to "Gerrit-Managers" [14:52:22] ottomata ^^ [14:52:25] (03PS2) 10Krinkle: utils: Add zuul-layout shaker [integration/config] - 10https://gerrit.wikimedia.org/r/501812 [14:54:09] _joe_: the config predates php72 use in prod. the default for jobs not caring about php support (e.g. sqlite, vendor, composertest) use php70 defacto. The extra php matrix is only mixed in on +2 for quicker turn around. [14:54:16] (03CR) 10jerkins-bot: [V: 04-1] utils: Add zuul-layout shaker [integration/config] - 10https://gerrit.wikimedia.org/r/501812 (owner: 10Krinkle) [14:54:19] also, older branches don't support php72 [14:54:37] so it's not straight forward in a wilderness of hundreds of regexes to make this work the way we want [14:54:50] <_joe_> Krinkle: right [14:54:56] but I've made some progress past weeks - https://gerrit.wikimedia.org/r/#/q/owner:Krinkle+project:integration/config+message:php72 [14:55:29] paladox: hm, ok, can I get in that? should I make a phab ticket? [14:55:53] ottomata you can, though yes it needs a task *i think*. [14:56:08] (03PS3) 10Krinkle: utils: Add zuul-layout shaker [integration/config] - 10https://gerrit.wikimedia.org/r/501812 [14:56:14] k thx [14:57:59] (03CR) 10jerkins-bot: [V: 04-1] utils: Add zuul-layout shaker [integration/config] - 10https://gerrit.wikimedia.org/r/501812 (owner: 10Krinkle) [14:59:07] (03PS1) 10Krinkle: Use php72 instead of php70 (5) - for mw postgres job [integration/config] - 10https://gerrit.wikimedia.org/r/502807 [15:02:36] 10Release-Engineering-Team (Backlog), 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10fsero) 05Open→03Resolved mwdebug2001,2 disk has been increased and VMs reimaged and pooled back, so this should be good to go now. While doing this i faced the sam... [15:05:12] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10Jdforrester-WMF) [15:06:06] <_joe_> paladox: I think repo owners (== people with +2) should be able to override jenkins vote [15:06:23] <_joe_> ottomata: we should open a bug about this ^^ instead [15:07:01] note if the "remove reviewers" right is not granted to project owners :) [15:07:05] *not [15:07:08] Actually [15:07:12] that's the fix [15:07:52] (03PS1) 10Paladox: Allow Project Owners to remove reviewers [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/502813 [15:08:01] _joe_ ottomata ^^ [15:09:04] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "This would allow repo owners to override reviews by CI when needed. I think they should have that right and that responsibility." [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/502813 (owner: 10Paladox) [15:09:22] thcipriani would you be able to review ^^ please? :) [15:10:44] * thcipriani looks [15:11:21] <_joe_> thcipriani: since you're here, can you point me to your scripts used to build docker images on contint1001? [15:11:29] sure [15:11:51] <_joe_> context is https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/deploy/+/502812 [15:11:58] https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/fabfile.py#57 [15:12:04] <_joe_> it's a breaking change and I want to fix downstream usages [15:12:23] <_joe_> ack, thanks [15:13:03] thanks thcipriani [15:16:35] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] "Looks like mediawiki has this on mediawiki. Seems like a necessary function at times (unfortunately)." [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/502813 (owner: 10Paladox) [15:17:07] thanks :) [15:18:29] (03PS1) 10Giuseppe Lavagetto: Conform docker-pkg invocation to docker-pkg 2.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/502816 [15:18:52] _joe_: We're running 7.0, 7.1 and 7.2 tests on Core on merge, but only 7.0 on test. I've got a patch re-directing that to 7.2, but we're still going to be running the slow-as-anything 7.0 and 7.1 suites until we drop support for those versions. :-( [15:19:04] <_joe_> sure [15:20:16] twentyafterfour: I think T220563 can be marked as Resolved without a backport (broken code since 2003, caused by on-wiki content, now fixed in master). [15:20:16] T220563: Undefined index: june in DateFormatter.php (makeIsoMonth) - https://phabricator.wikimedia.org/T220563 [15:21:26] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10Daimona) [15:24:49] <_joe_> thcipriani: heads up - I'm deploying docker-pkg 2.0.0, if you want to keep your fabfile working, you need to merge https://gerrit.wikimedia.org/r/502816 [15:24:54] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Yurik) @Milimetric I think the removal of Graphoid will be far more difficult than just keeping it. If you remove it, y... [15:25:33] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10Jdforrester-WMF) [15:25:36] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10thcipriani) Can we add back both @Legoktm and @QChris to Gerrit Administrators? Seems like they were doing work th... [15:25:40] * thcipriani does want phab file to keep working [15:25:47] er..fab [15:25:52] phabricator has broken me [15:25:58] :-) [15:25:59] <_joe_> indeed [15:26:08] <_joe_> it broke your spirit first, then your spelling [15:26:19] lol [15:26:30] (03CR) 10Thcipriani: [C: 03+2] "Thanks for the update!" [integration/config] - 10https://gerrit.wikimedia.org/r/502816 (owner: 10Giuseppe Lavagetto) [15:27:06] my spelling was always the weaker of the two, so who knows which went first. [15:27:44] <_joe_> thcipriani: I have one word for you: "arcanist" [15:27:55] <_joe_> you had to use that shit extensively [15:27:56] (03Merged) 10jenkins-bot: Conform docker-pkg invocation to docker-pkg 2.0.0 [integration/config] - 10https://gerrit.wikimedia.org/r/502816 (owner: 10Giuseppe Lavagetto) [15:28:25] :D [15:32:34] twentyafterfour: Never mind, Krinkle is deploying a back-port for it anyway. [15:33:03] Yeah, I figured might as well. It's not technically a regression in the code. Someone edited a wiki page, that's all. [15:33:09] but might as well dampen the noise [15:34:27] (03CR) 10Jforrester: [C: 03+1] Use php72 instead of php70 (5) - for mw postgres job [integration/config] - 10https://gerrit.wikimedia.org/r/502807 (owner: 10Krinkle) [15:35:12] Krinkle: I was just going to let it be the first bug fixed in 1.34.0. ;-) [15:35:38] oh may, are we pencils down already? [15:37:57] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10Krinkle) [15:38:37] Krinkle: Yup, wmf.25 was the last 1.33.0 alpha. [15:39:04] But REL1_33 hasn't been cut yet because Mukunda was rather busy with a broken train yesterday and I don't know how to do it for him. :-( [15:40:01] so with that, there's only the one blocker left iiuc? [15:42:36] apergos: There are no blockers left, except we need to run `scap clean` as the debug servers are full. [15:42:45] ah right [15:42:53] and it was acting up for some reason [15:43:08] Well, someone broke it by disabling HTTP Auth with gerrit. ;-) [15:43:44] So now it's been monkey-patched to not try to do that half of its job (gerrit branch cleanup), but we still need to run it for the first half (deployment server cleanup) [15:44:11] And however keen I am, I think that this is a bad time to take initiative and run it myself. :-) [15:44:17] ayup [15:44:23] (03PS6) 10Ejegg: DonationInterface tests run on PHP7 and MediaWiki 1.31 [integration/config] - 10https://gerrit.wikimedia.org/r/499334 [15:46:58] 10Release-Engineering-Team (Kanban), 10Multimedia, 10SDC Engineering, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), and 3 others: Jenkins job to run core tests against commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T220621 (10Cparle) [15:50:13] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T206678 (10Kipod) [15:54:00] 10Release-Engineering-Team (Kanban), 10Multimedia, 10SDC Engineering, 10MW-1.34-notes (1.34.0-wmf.1; 2019-04-16), and 3 others: Jenkins job to run core tests against commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T220621 (10zeljkofilipin) p:05Triage→03Normal [16:24:11] (03CR) 10Krinkle: Establish codehealth pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [16:26:20] (03PS9) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [16:27:00] Krinkle: Do you think T220623 should be a train blocker? [16:27:01] T220623: Unable to view certain pages on incubator.wikimedia.org (Fatal error: operator not supported) - https://phabricator.wikimedia.org/T220623 [16:27:19] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [16:28:01] James_F: No, not new. [16:28:46] OK. [17:02:53] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) I could not reproduce on my local machine given a basic test script which creates several heads: ` lang=python import git import logging logging.bas... [17:22:30] 10Project-Admins: Create a tag for new Community Tech project 'Night Mode - https://phabricator.wikimedia.org/T220571 (10Aklapper) This is basically {T218800} / {T199634} / {T206094}, I guess? [17:33:19] (03PS10) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [17:35:02] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [17:39:58] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, and 2 others: [Epic] ORES should use a git large file plugin for storing serialized binaries - https://phabricator.wikimedia.org/T171619 (10Halfak) 05Open→03Resolved a:03Halfak Seems like this is done. [17:40:05] 10Release-Engineering-Team (Kanban), 10Multimedia, 10SDC Engineering, 10Multimedia-Current-Work, and 2 others: Jenkins job to run core tests against commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T220621 (10Aklapper) [17:40:08] 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Security: Reenable l10update in production - https://phabricator.wikimedia.org/T220481 (10matmarex) Note also T158360 which proposes that LocalisationUpdate should be undeployed. [17:42:25] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10mmodell) I ran it for version 17 yesterday and that seems to have worked. I'll clean up 18-20 today. [17:43:08] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [17:45:01] (03CR) 10Thcipriani: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [17:45:10] (03CR) 10jerkins-bot: [V: 04-1] post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [17:47:27] (03PS1) 10Umherirrender: [ActiveAbstract] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502842 [17:47:33] Krinkle: Can I steal some of your time to help with https://gerrit.wikimedia.org/r/c/integration/config/+/501687/ (node10 for mobile content service)? [17:47:40] (03PS4) 1020after4: post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [17:47:49] (03CR) 10jerkins-bot: [V: 04-1] post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [17:51:51] 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Security: Reenable l10update in production - https://phabricator.wikimedia.org/T220481 (10Jdforrester-WMF) I think the short answer to your question is "no", but it's not my call, sorry. [17:55:40] (03PS5) 10Thcipriani: post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) [17:55:53] (03CR) 10Thcipriani: post pipeline: fix ref filter (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [17:57:07] (03CR) 10jerkins-bot: [V: 04-1] post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [17:57:59] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Jade, 10Scoring-platform-team: Clone gerrit repo mediawiki/extensions/JADE to mediawiki/extensions/Jade - https://phabricator.wikimedia.org/T212180 (10mmodell) 05Open→03Resolved [17:58:08] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [18:01:00] (03PS6) 10Thcipriani: post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) [18:28:08] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [18:38:49] bunch of csp-report-only errors: https://logstash-beta.wmflabs.org/goto/fc2c59726afd61d3098c0d552e23ab04 ^^ [18:39:26] by the name that makes me guess they aren't important [18:39:34] s/important/worrisome/ [19:22:02] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10mmodell) [19:22:31] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) So I think we can just prune all references from time to time: ` rm .git/packed_refs find .git/refs/ -mindepth 1 -delete git fetch git pack-refs ` [19:28:58] greg-g: Yeah, CSP reports are mostly ignored right now, and don't break anything. [19:44:03] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10mmodell) [19:47:48] So… how do we actually make the REL1_33 branch? Can we use make-wmf-branch with some interesting config? [19:51:10] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) A fast way to delete all Zuul references: git fetch gerrit --prune +refs/zuul/*:refs/zuul/* And to prune stalled local branches: git fetch g... [19:52:24] James_F: make-release/branch.py [19:52:37] James_F: warning, I am not 100% sure / untested etc :] [19:52:43] Ha. [19:55:19] Hmm, but that uses HTTP Auth which is disabled? [19:55:39] :/ [19:56:00] PROBLEM - Free space - all mounts on integration-slave-docker-1048 is CRITICAL: CRITICAL: integration.integration-slave-docker-1048.diskspace._srv.byte_percentfree (<10.00%) [20:00:01] Where does make-wmf-branch get the repos it should branch? [20:00:12] It's configured in the repo. [20:00:22] bawolff2: any idea why phan is playing up on https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/502191/ ? [20:00:34] paladox: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/release/+/master/make-wmf-branch/config.json [20:00:35] it complains about Html::element but Html::openElement is what is being used [20:01:09] ah [20:01:12] thanks James_F [20:06:01] RECOVERY - Free space - all mounts on integration-slave-docker-1048 is OK: OK: All targets OK [20:08:54] hashar: Well, https://wikitech.wikimedia.org/wiki/How_to_perform_security_fixes#4:_Push_to_gerrit_&_make_release is definitely out of date. :-( [20:10:49] But AFAICT we want to run `python ./branch.py --core --bundle "*" --branchpoint b1b89784f6df33d02d91401eaacc4aeab6bbedee --core-version 1.33.0` if only HTTP Auth worked. [20:14:07] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Release MW 1.33 - https://phabricator.wikimedia.org/T220652 (10Jdforrester-WMF) [20:14:50] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 (10Jdforrester-WMF) [20:15:28] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Release MW 1.33.0-rc.0 - https://phabricator.wikimedia.org/T220654 (10Jdforrester-WMF) [20:16:53] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Jdforrester-WMF) [20:16:58] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 (10Jdforrester-WMF) [20:17:34] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Announce 1.33 "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T220655 (10Jdforrester-WMF) [20:27:34] !log create operations/software/gerrit/plugins/MassBranchCreation repository [20:27:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:30:46] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), 10User-zeljkofilipin: Minerva browser tests must be configured to run on the mobile domain - https://phabricator.wikimedia.org/T220658 (10Jdlrobson) [20:30:51] (03PS1) 10Jdlrobson: Run Minerva browser tests against the mobile domain [integration/config] - 10https://gerrit.wikimedia.org/r/502884 (https://phabricator.wikimedia.org/T220658) [20:31:11] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), 10User-zeljkofilipin: Minerva browser tests must be configured to run on the mobile domain - https://phabricator.wikimedia.org/T220658 (10Jdlrobson) p:05Triage... [20:33:03] (03PS3) 10Jdlrobson: Run Node.js browser tests in MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/502356 (https://phabricator.wikimedia.org/T220659) [20:34:55] (03PS4) 10Jdlrobson: Run Node.js browser tests in MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/502356 (https://phabricator.wikimedia.org/T220659) [20:34:59] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 (10Jdforrester-WMF) I've marked T218750 as a blocker, but alternatively we could take the changes @thcipriani made to `make... [20:46:38] (03PS11) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [20:52:15] (03PS12) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [20:59:25] PROBLEM - zuul_merger_service_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [21:06:46] ACKNOWLEDGEMENT - zuul_merger_service_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger amusso zuul-merger maintenance https://www.mediawiki.org/wiki/Continuous_integration/Zuul [21:12:41] RECOVERY - zuul_merger_service_running on contint1001 is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [21:28:08] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [21:28:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Audit jenkins-bot@wikimedia.org usage and migrate to a better system - https://phabricator.wikimedia.org/T220663 (10hashar) [21:30:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Audit jenkins-bot@wikimedia.org usage and migrate to a better system - https://phabricator.wikimedia.org/T220664 (10hashar) [21:31:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Audit jenkins-bot@wikimedia.org usage and migrate to a better system - https://phabricator.wikimedia.org/T220664 (10hashar) [21:39:46] PROBLEM - Free space - all mounts on integration-slave-docker-1040 is CRITICAL: CRITICAL: integration.integration-slave-docker-1040.diskspace._srv.byte_percentfree (<10.00%) [21:40:14] maintenance-disconnect-full-disks build 62596 integration-slave-docker-1040 (/srv: 100%): OFFLINE due to disk space [21:42:29] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 (10Paladox) You can still use the rest api, just use your LDAP password instead of a http password generated through the ui. [21:54:44] RECOVERY - Free space - all mounts on integration-slave-docker-1040 is OK: OK: All targets OK [22:00:12] maintenance-disconnect-full-disks build 62600 integration-slave-docker-1040: OFFLINE due to disk space [22:03:08] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [22:14:59] PROBLEM - Free space - all mounts on integration-slave-docker-1037 is CRITICAL: CRITICAL: integration.integration-slave-docker-1037.diskspace._srv.byte_percentfree (<10.00%) [22:24:59] RECOVERY - Free space - all mounts on integration-slave-docker-1037 is OK: OK: All targets OK [22:25:12] maintenance-disconnect-full-disks build 62605 integration-slave-docker-1040: OFFLINE due to disk space [22:34:53] 10Project-Admins: Create a tag for new Community Tech project 'Night Mode - https://phabricator.wikimedia.org/T220571 (10jmatazzoni) >>! In T220571#5101877, @Aklapper wrote: > This is basically {T218800} / {T199634} / {T206094}, I guess? Yes and no. Those are all precursors, I'd say. It's possible none of them... [22:36:57] !log Deleted faulty REL1_33 branches for the Timeless, Vector and Monobook skins; they were duplicates of the REL1_32 branches. [22:36:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:38:23] PROBLEM - Free space - all mounts on integration-slave-docker-1052 is CRITICAL: CRITICAL: integration.integration-slave-docker-1052.diskspace._srv.byte_percentfree (<11.11%) [22:48:22] RECOVERY - Free space - all mounts on integration-slave-docker-1052 is OK: OK: All targets OK [22:50:11] maintenance-disconnect-full-disks build 62610 integration-slave-docker-1040: OFFLINE due to disk space [23:15:12] maintenance-disconnect-full-disks build 62615 integration-slave-docker-1040: OFFLINE due to disk space [23:15:13] maintenance-disconnect-full-disks build 62615 integration-slave-docker-1048 (/srv: 100%): OFFLINE due to disk space [23:17:00] PROBLEM - Free space - all mounts on integration-slave-docker-1048 is CRITICAL: CRITICAL: integration.integration-slave-docker-1048.diskspace._srv.byte_percentfree (<10.00%) [23:17:02] (03PS2) 10Jdlrobson: Run Minerva browser tests against the mobile domain [integration/config] - 10https://gerrit.wikimedia.org/r/502884 (https://phabricator.wikimedia.org/T190710) [23:17:34] (03PS3) 10Jdlrobson: Run Minerva browser tests against the mobile domain [integration/config] - 10https://gerrit.wikimedia.org/r/502884 (https://phabricator.wikimedia.org/T190710) [23:27:01] RECOVERY - Free space - all mounts on integration-slave-docker-1048 is OK: OK: All targets OK [23:32:07] !log Manually created REL1_33 branches for the core, vendor, and tarball extensions and skins. Eurgh. T220653 [23:32:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:32:10] T220653: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 [23:32:23] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release, 10Patch-For-Review: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 (10Jdforrester-WMF) In the end I did it manually rather than hold this up. [23:33:34] (03Abandoned) 10Dduvall: WIP Provide configurable pipeline execution [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 (owner: 10Dduvall) [23:33:41] (03Abandoned) 10Dduvall: WIP Provide local system tests using Jenkins in Docker [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/500134 (owner: 10Dduvall) [23:36:19] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Prune /vendor for REL1_33 - https://phabricator.wikimedia.org/T220667 (10Jdforrester-WMF) [23:37:31] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.33-notes, 10MW-1.33-release: Prune /vendor for REL1_33 - https://phabricator.wikimedia.org/T220667 (10Jdforrester-WMF) Pinging @legoktm who, unlike me, apparently knows how to determine what things to prune here, given they did it for REL1_32. :-) [23:37:51] (03PS1) 10Dduvall: pipeline: Execution graph and contexts [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) [23:37:53] (03PS1) 10Dduvall: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) [23:37:56] (03PS1) 10Dduvall: pipeline: Provide a rickety but useful system test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 [23:38:34] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [23:38:36] (03CR) 10jerkins-bot: [V: 04-1] pipeline: Execution graph and contexts [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [23:39:13] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [23:39:52] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 (owner: 10Dduvall) [23:40:11] maintenance-disconnect-full-disks build 62620 integration-slave-docker-1040: OFFLINE due to disk space [23:40:11] maintenance-disconnect-full-disks build 62620 integration-slave-docker-1048: OFFLINE due to disk space [23:49:05] (03PS2) 10Dduvall: pipeline: Execution graph and contexts [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) [23:49:07] (03PS2) 10Dduvall: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) [23:49:09] (03PS2) 10Dduvall: pipeline: Provide a rickety but useful system test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 [23:49:10] (03PS1) 10Jforrester: MediaWiki: REL1_33 now exists [integration/config] - 10https://gerrit.wikimedia.org/r/502922 [23:50:00] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [23:50:25] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [23:51:27] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 (owner: 10Dduvall)