[01:16:53] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:42] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 47988 bytes in 0.494 second response time [01:33:43] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:54] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:35] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 47988 bytes in 0.784 second response time [02:06:26] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:44] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 47999 bytes in 0.574 second response time [02:11:15] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48634 bytes in 0.957 second response time [02:40:05] (03CR) 10Krinkle: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/500511 (owner: 10Phedenskog) [02:43:57] (03Merged) 10jenkins-bot: jjb: Updated Fresnel to 0.2.2 [integration/config] - 10https://gerrit.wikimedia.org/r/500511 (owner: 10Phedenskog) [03:10:33] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:36:59] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:55:54] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Browser-Tests, 10Patch-For-Review, and 2 others: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10Jdlrobson) p:05Unbreak!→03High Hey @zeljkofilipin I've found a way to get them passing https://gerrit.wik... [04:31:51] 10Phabricator, 10Release-Engineering-Team (Kanban): task-series broken because of error: invalid type "custom.release.date" - https://phabricator.wikimedia.org/T219192 (10greg) [05:21:47] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [06:13:57] !log gerrit: renamed group "scholarships" to "wikimedia-wikimania-scholarships". Made it owned by "Gerrit Managers" # T218864 [06:14:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:14:00] T218864: group-scholarships access for MarcoAurelio - https://phabricator.wikimedia.org/T218864 [06:21:13] <_joe_> hashar: so, we need to rebuild all the images that derive from jessie [06:21:45] <_joe_> including some stretch images as your npm-stretch image copies things from the npm image [06:21:48] <_joe_> (EWWW) [06:23:13] (03PS4) 10Giuseppe Lavagetto: Rebuild all jessie-based images for removal of backports, updates [integration/config] - 10https://gerrit.wikimedia.org/r/500382 (https://phabricator.wikimedia.org/T219748) [06:23:39] <_joe_> hashar: greg-g told me to bug you in the morning, we should merge ^^ and then rebuild the images [06:23:51] yeah [06:24:00] I cant deal with that upgrade this week sorry :/ [06:24:05] but definitely on my radar for next week [06:24:14] (gotta rebuild and compare) [06:24:34] _joe_: ^^^ ;-/ [06:24:54] <_joe_> ok, but I've seen various images being rebuilt already this week, I had to rewrite the patch [06:25:02] <_joe_> anyways [06:26:45] _joe_: the thing is that some images will not be rebuild. I also don't know which priority the jessie-backports packages had [06:26:48] good news [06:26:56] I talked about it yesterday night with tyler [06:27:07] and I got to write a proposal to basaically overall the way the CI containers are build [06:27:21] to avoid those kind of troubles (upgrade a base image and have absolutely no control on the impact to the child images) [06:27:32] <_joe_> why they won't rebuild? [06:28:14] <_joe_> this seems like a wall of FUD [06:28:18] <_joe_> I'll be honest [06:28:48] <_joe_> the backports packages we used are backported by us into our repos [06:28:54] <_joe_> so there should be no issue [06:29:10] <_joe_> if anything from backports was used in these images and is not backported, we want to know and amend [06:29:56] <_joe_> what I see is an untangled web of dependencies that are poorly managed in CI [06:30:05] <_joe_> in integrations/config I mean [06:30:25] <_joe_> so one option you have is to force people to indicate a maintainer of a CI image [06:30:36] <_joe_> and notify the maintainers when a rebuild of their image is needed [06:30:52] <_joe_> or, we do run some form of testing on the image when building? [06:31:09] <_joe_> avoiding using the layering docker offers seems pointless [06:32:05] <_joe_> TL;DR please let's talk about this, I'm not happy at all at how docker images are being managed in CI, and it's only going to get worse if we don't do something about it now [06:32:21] well [06:32:23] really [06:32:28] lets talk about that next week [06:32:34] <_joe_> sure [06:32:53] and no its not a FUD [06:33:39] <_joe_> well "some omages won't rebuild" is [06:33:59] right now, you are making me angry [06:34:07] it is broken by design [06:34:11] <_joe_> what is? [06:34:33] the whole dependency of images, none of them being reproducible and all being tightly coupled withe parent [06:34:57] dropping the jessie-backports component is probably fine on the ci-jessie image [06:35:05] which is the base one [06:35:14] but you have no guarantee has to what will happen to the child images [06:35:19] <_joe_> as I said, packages from backports have been reimported [06:35:30] <_joe_> and just to be clear [06:35:36] new version of packages will be installed, maybe some variable parameter would have changed or is not available again [06:35:40] <_joe_> you had to explicitly select a package from backports [06:35:42] <_joe_> no no wait [06:35:56] <_joe_> see? this is what I referred to as FUD. Let's go in order. [06:36:06] it is not a fud [06:36:09] it is a real issue [06:36:27] <_joe_> 1 - removal of backports. You only got packages from backports if you explicitly requested them with apt-get install -t jessie-backports [06:36:35] <_joe_> or if you pinned packages in apt [06:36:55] <_joe_> so I went through the images (admittedly quickly) and found no such case [06:37:12] OR [06:37:23] when a a package was only javailable in jessie-backports [06:37:34] <_joe_> those were reimported [06:37:41] <_joe_> if one is missing, we will import it too [06:37:46] <_joe_> but we need to know [06:37:51] <_joe_> hence to build those images [06:37:53] <_joe_> anyways [06:37:59] <_joe_> please let me finish [06:38:04] ok ok :) [06:38:41] <_joe_> 2 - Jessie doesn't receive non-security updates (so no possibility of parameter changes) since 1 year or so. Thus I can't see how that could break something, but I understand the caution [06:39:10] <_joe_> btw, when we build a new server, that has the same "non reproducibility" issue [06:39:34] <_joe_> so I'm not sure what can happen unless it's some problem with how the dockerfiles are written [06:39:41] <_joe_> e.g. using pip without versioning [06:40:04] <_joe_> so if /that/ is what you fear, that is solved with two things: [06:40:21] <_joe_> 1 - some linter that doesn't allow setting unpinned pip/npm installations [06:40:39] <_joe_> 2 - running some test from docker-pkg on the built image if you provide it [06:40:51] <_joe_> the test I mean [06:41:33] yup there are some example-run.sh scripts to **manually** test out images [06:41:40] but it is no run by ci :\ [06:41:55] <_joe_> hashar: I would say docker-pkg could run them [06:42:00] <_joe_> when it has built an image [06:42:22] some other images have a smoke test or an assertion directly in the Dockerfile [06:42:23] <_joe_> build => if test_image.sh is present, run it [06:42:34] so that if the smoke/assert fails, the image building fail :] [06:42:34] <_joe_> or some other name we can come up with [06:42:44] <_joe_> yeah let's move that to docker-pkg [06:42:53] as for "fear", really it is not feat [06:42:55] err [06:42:59] s/feat/fear/ [06:43:18] <_joe_> I think you were focusing your attention on the wrong part of the chain [06:43:20] that is real failure I encountered multiple times when rebuilding a parent image and all its childs [06:43:38] <_joe_> I wasn't contesting that you have found inconsistencies in how images work when rebuilt [06:43:49] <_joe_> but that's due to our writing dockerfiles incorrectly [06:44:03] <_joe_> if you do "pip install docker-py" you're guaranteed to be screwed soon [06:44:25] yes that and others [06:44:26] <_joe_> should we force people to use frozen-requirements.txt / package.json in their docker images? [06:45:07] <_joe_> so we need to do some form of review on the integration/config tree and impose some policies [06:45:08] but that is not the only trouble [06:45:20] docker: quibble-jessie lack python3-git, use pip3 instead [06:45:22] <_joe_> including people need to write proper control files :) [06:45:35] <_joe_> ? [06:45:35] that one was the package disappearing, we might want to import it in our apt as you suggested [06:45:41] <_joe_> oh sure [06:45:56] <_joe_> that's the kind of things we do expect as partial fallout [06:46:04] we also have php 7.x packages installed from a third party (sury.org who is also the DD for the php packages) [06:46:05] <_joe_> but only way to find out is to actually build the images [06:46:17] <_joe_> oh yeah, those change every week [06:46:22] its gpg release key has changed and required an update [06:46:43] <_joe_> but that would be a problem for a server or VM as well [06:46:45] as a side effect, the php7.1 / php7.2 have been updated to whatever latest version is provided by sury.org which is probably fine ;] [06:47:04] <_joe_> it is, given we rebuild their packages for production [06:47:06] but could had been not back compatible / causing a bug somewhere in the build [06:47:21] so [06:47:35] in the end that is why I am extremely careful when rebuilding all images :/ [06:47:42] <_joe_> anyways my point is, if we want to use docker images, that doesn't scrub away the burden of maintaining them [06:48:06] <_joe_> and that's ok, but the issue is with how we write dockerfiles for ci, not respecting the rules we have for production things [06:48:31] <_joe_> I understand why we do, but now you understand why we don't want anything not coming from debs and official ones in production [06:48:53] <_joe_> anyways, re: missing packages in backports [06:49:00] <_joe_> I'll work with John and fix those [06:49:06] yes I am sold on the Debian packages concept [06:49:23] <_joe_> as I can rebuild the images locally and see what fails [06:49:33] <_joe_> so we remove that roadblock for you people [06:49:46] <_joe_> and you can deal with the fallout in terms of pip/npm/whatever :) [06:49:49] but even your apt repo keeps changing anyway and with multiple repositories / the apt pin / different priorities, it is hard for me to understand the end result :/ [06:50:27] such as component/ci having several different packages and being at priority 1001. Which to me sound like that anytime a new package is added to that component it would end up taking priority [06:51:22] but for sure I need to run tests automatically [06:51:46] and from discussion I had with Tyler, an idea was to just rebuild those images continuously [06:51:56] and have CI jobs to point to :latest [06:52:08] <_joe_> ahah [06:52:11] ;D [06:52:13] <_joe_> so do nightly bulds [06:52:17] yeah [06:52:27] <_joe_> btw the change that was merged to nightly builds breaks a lot of logic [06:52:30] <_joe_> I need to redo it [06:52:33] which still not solve the issue of context / universe ever changing or images not being reproducible [06:52:39] but at least if something break, we catch it earlier [06:52:52] <_joe_> yes, I think I suggested you to do something similar in the past [06:53:01] yeah we talked about it already [06:53:08] <_joe_> also, again, ask for more discipline in writing those control files :) [06:53:19] but my brain is slow at processing stuff, I gotta think a lot [06:53:21] :/ [06:53:41] <_joe_> and yes, it would be great if we had a way to run docker-pkg from CI when we change one docker file in integration/config [06:53:43] anyway this morning when I woke up, I think my first thought has been: but how do we garbage collect all those nightly images from the registry [06:53:49] <_joe_> and / or production-images [06:53:54] <_joe_> hashar: HA! [06:54:17] <_joe_> hashar: the new registry fabian is working on will have proper delete capabilities [06:54:46] with some kind of expiry maybe? [06:55:00] <_joe_> we can write some gc cronjob I guess [06:55:18] <_joe_> "keep only the latest 10 nightly builds" [06:55:23] that is surely solvable [06:59:37] (03CR) 10Hashar: "Giuseppe and I talked about it this morning for breakfast. That might have a bunch of side effects such as:" [integration/config] - 10https://gerrit.wikimedia.org/r/500382 (https://phabricator.wikimedia.org/T219748) (owner: 10Giuseppe Lavagetto) [07:02:13] 10Release-Engineering-Team (Kanban), 10commit-message-validator, 10Patch-For-Review: commit-message-validator fail when an extra Hosts: header is mentioned - https://phabricator.wikimedia.org/T219629 (10hashar) 05Open→03Resolved @aborrero eventually the fix will be released (need a git tag then the packa... [07:06:44] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:32:06] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10WMDE-leszek) Thanks for the answers @greg. I understand @Add... [07:32:31] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10WMDE-leszek) p:05Unbreak!→03High [07:46:29] 10Phabricator: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX - https://phabricator.wikimedia.org/T209463 (10hashar) @Aklapper thank you for the js snippet, that is indeed what I had in mind. Maybe I should fill that as a feature request for pherrit instead. [08:19:42] 10Continuous-Integration-Infrastructure, 10Wikimedia-General-or-Unknown, 10Core Platform Team (Code Health (TEC13)), 10Core Platform Team Kanban (Doing), and 4 others: Deprecate/obsolete $wgWikimediaJenkinsCI - https://phabricator.wikimedia.org/T200650 (10hashar) [10:19:15] 10Continuous-Integration-Infrastructure, 10Shinken, 10Patch-For-Review: Shinken keeps alerting about long gone instances - https://phabricator.wikimedia.org/T218146 (10GTirloni) a:05GTirloni→03None [10:53:35] 10Phabricator, 10Operations, 10Traffic: Make phame cacheable - https://phabricator.wikimedia.org/T219978 (10ema) [10:53:42] 10Phabricator, 10Operations, 10Traffic: Make phame cacheable - https://phabricator.wikimedia.org/T219978 (10ema) p:05Triage→03Normal [10:56:07] (03PS4) 10Tim Eulitz: Create selenium-daily-beta-AdvancedSearch Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [11:56:10] Hi [11:56:34] can someone check https://gerrit.wikimedia.org/r/#/c/499530/ [12:38:31] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [12:40:58] caused by gerrit restart. no worries. self-fixing [12:43:49] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:18:26] can i search for tikcets in phab by "is public or NDA" ? [13:18:47] clicking "edit query" in a search and see a lot to search by but not that one [13:19:44] oh.. maybe by tag and select WMF-NDA [13:20:45] yep, got what i wanted [13:26:08] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Add my private email address to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/500178 (owner: 10Lucas Werkmeister (WMDE)) [13:35:17] 10Continuous-Integration-Infrastructure, 10Shinken, 10Patch-For-Review: Shinken keeps alerting about long gone instances - https://phabricator.wikimedia.org/T218146 (10hashar) I think it is solved? At least I have not received ghost notifications this week. [13:50:07] 10Release-Engineering-Team, 10Operations: mwdebug2001 "/" almost full - https://phabricator.wikimedia.org/T219989 (10Marostegui) [14:08:42] (03CR) 10Zfilipin: "This change is ready for review." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [14:15:53] (03PS5) 10Tim Eulitz: Create selenium-daily-beta-AdvancedSearch Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [14:40:13] 10Phabricator, 10Operations, 10Traffic: Make phame cacheable - https://phabricator.wikimedia.org/T219978 (10ema) [15:01:20] PROBLEM - Puppet errors on deployment-cache-text05 is CRITICAL: CRITICAL: 7.87% of data above the critical threshold [3.0] [15:03:50] 10Release-Engineering-Team, 10Operations: mwdebug2001 "/" almost full - https://phabricator.wikimedia.org/T219989 (10greg) [15:03:53] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review, 10User-zeljkofilipin: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10greg) [15:03:57] 10Release-Engineering-Team, 10Operations: mwdebug2001 "/" almost full - https://phabricator.wikimedia.org/T219989 (10thcipriani) This is related to T218783 [15:04:29] thcipriani: hah ^ [15:04:40] greg-g: jinx. [15:04:45] :) [15:54:32] 10Release-Engineering-Team, 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10jcrespo) [16:01:15] is there an official process for getting a new version of a CI docker merged->updated->usable? [16:02:14] (I guess I should qualify: by someone who doesn't have +2 and doesn't want to know what happens after +2) [16:04:18] in any case, https://gerrit.wikimedia.org/r/c/integration/config/+/500719 is waiting for whatever needs to happen there, if someone can do something! :) [16:10:31] PROBLEM - Host deployment-db03 is DOWN: CRITICAL - Host Unreachable (172.16.5.23) [16:14:00] Train question: Is stuff tagged (1.33.0-wmf.24; 2019-04-02) deployed on test wikis on that date, and then released later? Or deployed everywhere? [16:18:42] mvolz: Usually it's tagged and to group0 the same day (barring any major issues) [16:19:03] .24 looks to be on group0 already [16:22:55] ok, I want to fix something that's in group 0 before it gets to group2, so that's tomorrow, correct? [16:23:06] https://wikitech.wikimedia.org/wiki/Deployments/One_week [16:23:10] bah [16:25:05] thanks [16:26:51] Yeah, I think before 1900 UTC on Thursday [16:27:14] It doesn't affect any of the canary Wikipedias, but it affects at least one of the others, it depends on the config [16:27:28] tnx [16:27:40] If there's a relevant task, you can tag/block it on phab, and leave a comment too [16:28:31] there's this one which was the original phab task where it broke: https://phabricator.wikimedia.org/T219510 [16:29:47] what kind of tag? [16:39:49] 10Continuous-Integration-Config, 10Operations, 10Patch-For-Review, 10User-zeljkofilipin: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10Krinkle) a:05Krinkle→03MoritzMuehlenhoff OK. Looks like the image will alread... [16:45:15] mvolz: may want to add as subtask to https://phabricator.wikimedia.org/T206678 to make sure the train won't roll forward just in case it isn't deployed yet or something happens, as a reminder for REleng :) [16:46:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T206678 (10Mvolz) [16:46:46] Krinkle: Done, but I'm not actually sure it's a blocker [16:47:13] the worst that will happen is that citoid will be disabled on gu wiki which, to be frank, has happened before ^-^ [17:51:19] RECOVERY - Puppet errors on deployment-cache-text05 is OK: OK: Less than 1.00% above the threshold [2.0] [17:51:33] zeljkof: are you still around? Still having the issues from yesterday and wanted to check you got Greg's ping https://phabricator.wikimedia.org/T219920 [17:52:12] Ruby selenium tests cannot login anymore [18:22:14] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 3.37% of data above the critical threshold [3.0] [18:23:49] 10Continuous-Integration-Config, 10OOUI, 10Performance: Add a Fresnel task to OOUI repo to check performance impacts of changes as they get made - https://phabricator.wikimedia.org/T220024 (10Jdforrester-WMF) [18:28:55] 10Continuous-Integration-Config, 10OOUI, 10Performance-Team (Radar): Add a Fresnel task to OOUI repo to check performance impacts of changes as they get made - https://phabricator.wikimedia.org/T220024 (10Krinkle) [18:29:03] 10Continuous-Integration-Config, 10OOUI, 10Performance-Team (Radar): Add a Fresnel task to OOUI repo to check performance impacts of changes as they get made - https://phabricator.wikimedia.org/T220024 (10Krinkle) Awesome :) [18:34:35] 10Release-Engineering-Team, 10Browser-Tests, 10MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), 10Patch-For-Review, and 3 others: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10Jdlrobson) I'm disabling these tests as a short term measure, but that's not an accepta... [18:44:04] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10greg) The right answer here is to add Kunal to the Gerrit-Mangers group, I believe. Correct, @hashar ? [18:53:11] PROBLEM - Host deployment-db04 is DOWN: CRITICAL - Host Unreachable (172.16.5.5) [19:10:28] 10Release-Engineering-Team (Kanban), 10Trash: test - https://phabricator.wikimedia.org/T220027 (10mmodell) [19:11:19] 10Release-Engineering-Team (Kanban), 10Trash: test - https://phabricator.wikimedia.org/T220027 (10mmodell) 05Open→03Resolved [19:34:00] jdlrobson: on the phone, sorry saw your ping but ran out of time today, will take a look tomorrow [19:49:36] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10Jdforrester-WMF) [19:59:09] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10RobH) a:05greg→03hashar [20:05:52] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade webdriverio to version 5 - https://phabricator.wikimedia.org/T213268 (10Jdforrester-WMF) [20:06:59] 10Release-Engineering-Team (Kanban): Spike in DBTransactionError following 1.33.0-wmf.24 group1 promotion - https://phabricator.wikimedia.org/T220037 (10dduvall) [20:09:23] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10hashar) >>! In T219086#5082664, @greg wrote: > The right answer here is to add Kunal to the Gerrit-Managers group,... [20:35:42] (03CR) 10Giuseppe Lavagetto: "FTR, I rebuilt all these images locally and they all rebuild, minus the lintr image that fails with" [integration/config] - 10https://gerrit.wikimedia.org/r/500382 (https://phabricator.wikimedia.org/T219748) (owner: 10Giuseppe Lavagetto) [21:02:38] (03PS10) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [21:06:11] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [21:06:15] (03CR) 10jerkins-bot: [V: 04-1] Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [21:20:11] 10Continuous-Integration-Config, 10OOUI, 10Performance-Team (Radar): Add a Fresnel task to OOUI repo to check performance impacts of changes as they get made - https://phabricator.wikimedia.org/T220024 (10Volker_E) p:05Triage→03Normal [21:23:19] (03PS11) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [21:26:03] (03CR) 10Effie Mouzeli: Add --canary-wait-time flag (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [21:26:08] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [21:26:10] (03CR) 10jerkins-bot: [V: 04-1] Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [21:28:23] (03PS12) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [21:31:11] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [22:20:13] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Release Pipeline, 10local-charts: Define a base docker-pkg template and .pipeline/blubber.yaml for mediawiki/core - https://phabricator.wikimedia.org/T218360 (10brennen) a:03brennen [22:22:46] 10Release-Engineering-Team (Kanban), 10Browser-Tests, 10MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), 10Patch-For-Review, and 3 others: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10Jdlrobson) a:03zeljkofilipin So I've disabled quite a few tests (over 20) on...