[00:55:35] (03CR) 10AndyRussG: "Cool, thanks for working on this! Looking forward to the update mentioned..." (033 comments) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/664919 (owner: 10Jgleeson) [06:32:33] 10Phabricator, 10Release-Engineering-Team (Development services): Clarify purpose of / disable GerritToPhabBot user account in Phabricator - https://phabricator.wikimedia.org/T243597 (10Aklapper) 05Open→03Resolved [06:32:48] 10Phabricator, 10Release-Engineering-Team (Development services): Clarify purpose of / disable GerritToPhabBot user account in Phabricator - https://phabricator.wikimedia.org/T243597 (10Aklapper) Account disabled. Let's see if someone/something will yell. [07:01:31] 10Beta-Cluster-Infrastructure, 10Discovery-Search: Search broken on Beta Wikimedia Commons - https://phabricator.wikimedia.org/T275975 (10Aklapper) [07:06:41] 10Project-Admins: Create User-Inductiveload project - https://phabricator.wikimedia.org/T275929 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #user-inductiveload has been created: https://phabricator.wikimedia.org/project/view/5221/ Feel free to [watch the project](https://www.media... [07:06:44] 10Phabricator (Upstream), 10Upstream: Per-user projects for personal work in progress tracking - https://phabricator.wikimedia.org/T555 (10Aklapper) [08:04:42] 10Project-Admins, 10WMSE: Move "Magnus Sälgö" and "so9q" to top level projects - https://phabricator.wikimedia.org/T272137 (10Lokal_Profil) looks like you got to the bottom of this on the discussion page. Let me know if you need any #WMSE input. [08:05:22] 10Project-Admins, 10WMSE (IT): Move "Magnus Sälgö" and "so9q" to top level projects - https://phabricator.wikimedia.org/T272137 (10Lokal_Profil) [08:49:57] hashar (or someone else who can merge CI config patches): if you have time to merge https://gerrit.wikimedia.org/r/c/integration/config/+/667135 (ping me if you need more details) [08:54:51] 10Project-Admins: Create dedicated project tag for Spicerack, and automatically add #sre-tools to its tasks - https://phabricator.wikimedia.org/T275488 (10Aklapper) 05Open→03Resolved a:03Aklapper * `#spicerack` additional hashtag removed from `#sre-tools` at https://phabricator.wikimedia.org/project/edit/1... [09:32:58] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10Pywikibot, and 2 others: cirrussearch-backend-error at beta cluster - https://phabricator.wikimedia.org/T276038 (10Xqt) [09:33:27] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10Pywikibot, and 2 others: cirrussearch-backend-error at beta cluster - https://phabricator.wikimedia.org/T276038 (10Xqt) [09:40:45] 10Project-Admins, 10WMSE (IT): Move "Magnus Sälgö" and "so9q" to top level projects - https://phabricator.wikimedia.org/T272137 (10Aklapper) I archived #so9q at https://phabricator.wikimedia.org/project/profile/5146/ and [resolved its open tasks]( https://phabricator.wikimedia.org/maniphest/?ids=272285,272283,... [10:04:49] (03CR) 10Hashar: [C: 03+2] "Indeed thanks to the new "workdir" parameter :) Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/667135 (owner: 10Gehel) [10:06:40] (03Merged) 10jenkins-bot: Fix pom path for xgboost project. [integration/config] - 10https://gerrit.wikimedia.org/r/667135 (owner: 10Gehel) [10:07:25] (03CR) 10Hashar: [C: 03+2] Add ContentTransfer, MergeArticles, BSPlayer, BSFRConnector [integration/config] - 10https://gerrit.wikimedia.org/r/666326 (owner: 10Pwirth) [10:09:13] (03Merged) 10jenkins-bot: Add ContentTransfer, MergeArticles, BSPlayer, BSFRConnector [integration/config] - 10https://gerrit.wikimedia.org/r/666326 (owner: 10Pwirth) [10:10:36] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/666326 (owner: 10Pwirth) [10:13:43] (03CR) 10Hashar: [C: 03+2] "Jobs created" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [10:15:05] (03PS5) 10Hashar: Zuul: [mediawiki/services/function-schemata] Use bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [10:15:28] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/services/function-schemata] Use bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [10:17:43] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/function-schemata] Use bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [10:19:19] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/665419 (owner: 10Cory Massaro) [10:54:11] (03CR) 10Hashar: "I am a bit surprised by this change, we have a few jobs invoking a specific tox environment and surely they would have been broken long ag" [integration/config] - 10https://gerrit.wikimedia.org/r/663212 (https://phabricator.wikimedia.org/T274370) (owner: 10David Caro) [11:24:08] (03PS1) 10MSantos: Zuul: [mediawiki/services/kartotherian] remove tilerator publish pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/667569 (https://phabricator.wikimedia.org/T275845) [11:25:24] (03PS2) 10MSantos: Zuul: [mediawiki/services/kartotherian] remove tilerator publish pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/667569 (https://phabricator.wikimedia.org/T275845) [11:26:41] (03CR) 10jerkins-bot: [V: 04-1] Zuul: [mediawiki/services/kartotherian] remove tilerator publish pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/667569 (https://phabricator.wikimedia.org/T275845) (owner: 10MSantos) [11:29:50] (03PS3) 10MSantos: Zuul: [mediawiki/services/kartotherian] remove tilerator publish pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/667569 (https://phabricator.wikimedia.org/T275845) [11:36:03] 10Project-Admins: Request for project MediaWiki-extensions-SemanticQueryInterface - https://phabricator.wikimedia.org/T138603 (10Aklapper) @Vedmaka: Could you please answer the last comment? Thanks a lot in advance! :) [12:49:38] (03CR) 10Hashar: [C: 03+2] "I have deleted the jobs ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/667569 (https://phabricator.wikimedia.org/T275845) (owner: 10MSantos) [12:51:03] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/kartotherian] remove tilerator publish pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/667569 (https://phabricator.wikimedia.org/T275845) (owner: 10MSantos) [12:51:30] 10Beta-Cluster-Infrastructure, 10Discovery-Search: Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10kostajh) [12:52:43] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10Pywikibot, and 2 others: cirrussearch-backend-error at beta cluster - https://phabricator.wikimedia.org/T276038 (10kostajh) [12:52:46] 10Beta-Cluster-Infrastructure, 10Discovery-Search: Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10kostajh) [14:46:20] Hello all! I'm tidying up puppet in deployment-prep and there are two broken VMs (deployment-sca02 and deployment-sca01) that depend on a puppet profile that no longer exists, profile::recommendation_api. Can I delete those two VMs? [14:46:33] They were created by Marco who is no longer with the foundation as far as I know. [14:54:52] 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): broken puppet on deployment-sca01.deployment-prep.eqiad1.wikimedia.cloud and deployment-sca02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T276102 (10Andrew) [14:56:47] 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): broken puppet on deployment-sca01.deployment-prep.eqiad1.wikimedia.cloud and deployment-sca02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T276102 (10Majavah) 356ca06/T220235 suggests that yes, but not an expert [15:03:35] 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): Broken puppet on deployment-puppetdb04.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T276103 (10Andrew) [15:11:18] 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): Broken puppet on deployment-puppetdb04.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T276103 (10Andrew) 05Open→03Resolved a:03Andrew I think I sorted this out. [15:25:17] (03PS1) 10Addshore: Add {name}-node10-browser-webdriver-docker [integration/config] - 10https://gerrit.wikimedia.org/r/667623 (https://phabricator.wikimedia.org/T222200) [15:26:12] (03Abandoned) 10Addshore: Change Dockerfile and wikidata.yml to support running browser tests for the query service gui [integration/config] - 10https://gerrit.wikimedia.org/r/511308 (https://phabricator.wikimedia.org/T222200) (owner: 10Noa wmde) [15:26:33] (03CR) 10Awight: "Good idea!" [integration/config] - 10https://gerrit.wikimedia.org/r/667623 (https://phabricator.wikimedia.org/T222200) (owner: 10Addshore) [15:28:09] (03CR) 10jerkins-bot: [V: 04-1] Add {name}-node10-browser-webdriver-docker [integration/config] - 10https://gerrit.wikimedia.org/r/667623 (https://phabricator.wikimedia.org/T222200) (owner: 10Addshore) [15:32:37] (03CR) 10Addshore: "CI failure:" [integration/config] - 10https://gerrit.wikimedia.org/r/667623 (https://phabricator.wikimedia.org/T222200) (owner: 10Addshore) [15:33:15] Do I need to add the template and then use of templated jobs in 2 separate config changes? [15:33:20] I have a lingering memory to that being true [16:04:10] addshore: Yes. [16:12:31] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Legoktm) [16:19:10] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10MPhamWMF) [16:31:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Scap, 10Patch-For-Review: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10thcipriani) >>! In T266055#6820394, @dancy wrote: >>>! In T2... [16:56:50] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10MPhamWMF) [17:17:04] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:21:46] (03CR) 10MSantos: "> Patch Set 3: Code-Review+2" [integration/config] - 10https://gerrit.wikimedia.org/r/667569 (https://phabricator.wikimedia.org/T275845) (owner: 10MSantos) [17:27:47] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:31:02] when jenkins-bot says -1 but is NOT called jerkins-bot ... [17:31:12] that means there is an issue with jenkins itself it seems [17:31:36] When did it do that? [17:32:16] RhinosF1: eh, i think that was just me, but can you see a reason why this one fails? https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster-docker/20803/console [17:32:49] RhinosF1: oooh, of course I see it now right after asking :) [17:32:52] just had to ask [17:33:01] " Node 'mwmaint2002.codfw.wmnet' is already defined" [17:33:01] mutante: Node 'mwmaint2002.codfw.wmnet' is already defined (file: /srv/workspace/puppet/manifests/site.pp, line: 1867); cannot redefine (file: /srv/workspace/puppet/manifests/site.pp, line: 1872) [17:33:11] RhinosF1: ack, thank you:) fixing [17:33:16] But sometimes asking out loud makes it shout out loud at you [17:33:28] exactly, it was basically doing that on purpose :) [17:33:45] :) [17:35:04] That's the good thing about being remote, you can't hear me thinking out loud [17:37:35] 10Continuous-Integration-Config, 10Toolforge, 10cloud-services-team (Kanban): Set up CI for cloud/toolforge/buildpacks repository - https://phabricator.wikimedia.org/T265685 (10Legoktm) a:05Legoktm→03None [17:39:27] 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): broken puppet on deployment-eventlog05.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T276122 (10Andrew) [17:43:10] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dduvall) >>! In T273521#6854437, @Legoktm wrote: > The restricted/ namespace is now li... [17:43:30] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: legoktm can't build CI docker images without using root because he's no longer in contint-admins - https://phabricator.wikimedia.org/T275731 (10hashar) I would rather have cherry picked people that knows about docker-pkg / CI. But I guess it is fi... [17:45:51] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10dduvall) In other words, override the value of [[ https://gerrit.wikimedia.org/r/plugi... [17:50:03] 10Release-Engineering-Team-TODO, 10MW-on-K8s, 10Release Pipeline, 10serviceops, 10Patch-For-Review: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10Legoktm) >>! In T273521#6871522, @dduvall wrote: > PipelineLib currently calls out to... [17:56:09] marxarelli: is docker-pusher used on contint2001 as well? I noticed the profile is enabled on that host too [17:57:03] it is, yeah [17:58:01] ok, and contint should *not* be able to push to restricted/, correct? [17:58:06] correct! [17:58:32] we don't want the credentials exposed to the contint hosts at all [17:58:53] credentials for restricted/ that is [17:59:24] I'll put the "ci-build" credentials there [17:59:42] which is plain push access, but not to restricted/ (read or write) [17:59:52] perfect [18:00:44] just fyi, how the pushing of images works in pipelinelib https://gerrit.wikimedia.org/r/plugins/gitiles/integration/pipelinelib/+/refs/heads/master/src/org/wikimedia/integration/PipelineRunner.groovy#282 [18:02:04] so is https://integration.wikimedia.org/ci/credentials/store/system/domain/service-pipeline/credential/docker-registry-uploader/ no longer used? [18:03:44] hmm... i'm not really aware of that credential or whether it has ever been used :/ [18:03:51] longma: ^ do you know? [18:03:59] oh whoops. meeting. bbiab :) [18:10:21] (03CR) 10Ahmon Dancy: [C: 03+1] Expand context variables within map keys [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/666478 (owner: 10Dduvall) [18:12:03] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.32 deployment blockers - https://phabricator.wikimedia.org/T274936 (10jeena) 05Open→03Resolved [18:19:37] marxarelli: legoktm I don't see it used in either integration/config or pipelinelib [18:21:30] maybe it's for docker-pkg use or something. hash'ar would know [18:21:47] docker-pkg credentials are in /etc/docker-pkg/*.yaml [18:21:55] 10Gerrit, 10GitLab (Initialization), 10User-brennen: Provide Speed & Function rough numbers for our current Gerrit web traffic - https://phabricator.wikimedia.org/T275564 (10brennen) @Sergey.Trofimovsky.SF - this is followup from questions you raised during our first meeting the other day. Please let us kno... [18:23:01] nothing in the releng SAL for it at least [18:25:00] "This credential has not been recorded as used anywhere." [18:25:03] i'm just going to delete it [18:25:08] (03CR) 1020after4: [C: 03+2] "I got it working locally. I will follow up with some more current into for the README but this looks great to me!" [releng/phatality] - 10https://gerrit.wikimedia.org/r/667269 (owner: 10Cwhite) [18:25:36] !log deleting unused docker-registry-uploader jenkins credential [18:25:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:26:18] legoktm: ^ :) [18:27:23] :D [18:29:12] awesome, now the credentials are only in places that puppet can reach, which means rotation is really straightforward [18:29:38] excellent! [18:30:40] (03PS2) 10Dduvall: Expand context variables within map keys [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/666478 [18:30:44] added you as a reviewer on https://gerrit.wikimedia.org/r/c/operations/puppet/+/667682/ [18:30:58] * marxarelli looks [18:31:49] (03CR) 10Ahmon Dancy: [C: 03+1] Expand context variables within map keys [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/666478 (owner: 10Dduvall) [18:35:06] (03CR) 1020after4: [V: 03+2 C: 03+2] Upgrade Phatality to v7.10.0 compatibility [releng/phatality] - 10https://gerrit.wikimedia.org/r/667269 (owner: 10Cwhite) [18:35:11] (03CR) 10Dduvall: [C: 03+2] Expand context variables within map keys [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/666478 (owner: 10Dduvall) [18:36:26] (03Merged) 10jenkins-bot: Expand context variables within map keys [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/666478 (owner: 10Dduvall) [18:52:52] 10MediaWiki-Codesniffer, 10MediaWiki-General: Enforce usage of phpdoc's @api and @internal tags to indicate public and private structural elements - https://phabricator.wikimedia.org/T209394 (10Krinkle) Now that we have a [Stable interface policy](https://www.mediawiki.org/wiki/Stable_interface_policy) this be... [18:53:10] 10MediaWiki-Codesniffer, 10phan, 10MediaWiki-Core-Testing: Use sniff or phan rule to detect violations of Stable interface policy (@internal, @stable etc.) - https://phabricator.wikimedia.org/T209394 (10Krinkle) [18:54:02] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: map on how Gitlab is accessed - https://phabricator.wikimedia.org/T276144 (10thcipriani) [18:55:37] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Map of how Gitlab is accessed - https://phabricator.wikimedia.org/T276144 (10thcipriani) [18:55:38] 10Release-Engineering-Team, 10MW-on-K8s, 10Release Pipeline, 10serviceops-radar, 10Epic: Ensure MW PrivateSettings exist on releases server for production MW builds - https://phabricator.wikimedia.org/T276145 (10jeena) [18:57:18] https://grafana.wikimedia.org/d/000000321/zuul?orgId=1&from=1614020079042&to=1614624879042&viewPanel=25 [18:57:25] https://grafana.wikimedia.org/d/000000321/zuul?orgId=1 [18:57:47] ^ a lot of jobs for zuul and the "gate processing time" is marked as pink /elevated [18:58:03] it's still working, just a spike of new jobs [18:59:50] mutante: can you see what's in the queue [19:00:16] RhinosF1: https://integration.wikimedia.org/zuul/ [19:00:36] Majavah: I meant like titles or something [19:00:47] So you could tell if it's automated or just busy humans [19:03:01] I randomly clicked a few (there's a lot of MediaWiki patches) and they all seem like normal human patches [19:03:19] Don't see a lot from Amir1's wiki creation so I can't blame that [19:03:25] Bot** [19:03:39] What did I break now [19:03:51] Amir1: nothing [19:03:56] phew [19:03:58] * Amir1 hides [19:04:01] I'm saying it doesn't look like it's you [19:08:26] pretty sure it's just normal Monday activity [19:09:28] Nothing looks like it could be anything else [19:11:40] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10thcipriani) [19:12:47] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10thcipriani) [19:13:06] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10thcipriani) [19:15:46] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10thcipriani) The hope with this task is to try to surface additional concerns about the proposal so that we can open ssh c... [20:20:49] (03CR) 10Brennen Bearnes: "> Patch Set 2:" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/663159 (https://phabricator.wikimedia.org/T262975) (owner: 10AndyRussG) [20:29:28] the new docker-pusher credentials are in place on releases* and contint* hosts [20:29:53] (03CR) 10Brennen Bearnes: "> Cool, testing build now..." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/663159 (https://phabricator.wikimedia.org/T262975) (owner: 10AndyRussG) [20:46:47] 10Gerrit, 10Release-Engineering-Team, 10serviceops: Gerrit crashed due to out of Heap - https://phabricator.wikimedia.org/T225166 (10hashar) The issue was a memory leak I have found in Gerrit (T263008). It has been addressed in Gerrit 3.2.7 which we deployed in February 2020. [20:47:02] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Port map of how Gitlab is accessed - https://phabricator.wikimedia.org/T276144 (10thcipriani) [20:48:06] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Port map of how Gitlab is accessed - https://phabricator.wikimedia.org/T276144 (10wkandek) a:03Dzahn [21:00:48] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10thcipriani) [21:01:15] 10Release-Engineering-Team-TODO, 10serviceops, 10GitLab (Initialization), 10User-brennen: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10thcipriani) [21:02:40] 10Release-Engineering-Team-TODO, 10serviceops, 10GitLab (Initialization), 10User-brennen: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10thcipriani) Are there other subtasks (IP allocation or anything) that need seperate subtasks? [21:16:10] 10Release-Engineering-Team (Logspam), 10MediaWiki-Revision-backend, 10MediaWiki-User-management, 10Move-Files-To-Commons, and 3 others: CannotCreateActorException: Cannot create an actor for a usable name that is not an existing user: … (via SpecialImportFile) - https://phabricator.wikimedia.org/T275067 (1... [21:34:29] 10Beta-Cluster-Infrastructure: [betalabs] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Etonkovidova) [21:36:13] 10Beta-Cluster-Infrastructure: [betalabs] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Etonkovidova) [21:47:51] 10Beta-Cluster-Infrastructure: [betalabs] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Reedy) I don't think the regeneration scripts run on beta... But indeed, apparently zero captcha ` reedy@deployment-deploy01:~$ /usr/local/bin/mwscript extensions/ConfirmEdit/ma... [21:49:03] 10Beta-Cluster-Infrastructure: Beta SWIFT seems to be broken - https://phabricator.wikimedia.org/T276179 (10Reedy) [21:49:14] 10Beta-Cluster-Infrastructure: [betalabs] UnderflowException: Ran out of captcha images - https://phabricator.wikimedia.org/T276176 (10Reedy) [21:49:17] 10Beta-Cluster-Infrastructure: Beta SWIFT seems to be broken - https://phabricator.wikimedia.org/T276179 (10Reedy) [22:39:01] 10Release-Engineering-Team (Code Health), 10MediaWiki-extensions-FlaggedRevs, 10Code-Stewardship-Reviews: Code stewardship review: FlaggedRevs - https://phabricator.wikimedia.org/T185664 (10Proc) So... what's going on with this task (and FlaggedRevs in general)? [22:41:07] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10serviceops: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) [22:59:18] can maybe someone help me understand why https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/81767/console is failing? nothing in my change should affect that test [23:01:18] it's a flaky/broken browser test in MinervaNeue [23:01:30] 22:40:07 [Chrome 73.0.3683.75 linux #0-5] 1) Opening and closing the reference drawer Opening a nested reference [23:01:30] 22:40:07 [Chrome 73.0.3683.75 linux #0-5] Node is either not visible or not an HTMLElement [23:02:14] so the failure can be ignored? [23:03:41] maybe try if you can repeat it with recheck [23:04:58] mutante: ill try rebasing it to make sure that doesnt cause any issues too [23:05:30] seems good [23:07:41] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Port map of how Gitlab is accessed - https://phabricator.wikimedia.org/T276144 (10Dzahn) Which of the 2 VMs should get the traffic for http/https? Assuming gitlab1001 is fine and gitlab1002 isn't a we... [23:13:15] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: Port map of how Gitlab is accessed - https://phabricator.wikimedia.org/T276144 (10Dzahn) An issue here is that ATS normally speaks only https to backends. But this opens a bunch of questions that invo... [23:13:45] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Sergey.Trofimovsky.SF) Given the ratio between number of people that will be using git SSH access, and people that will l... [23:14:44] 10Beta-Cluster-Infrastructure, 10Discovery-Search (Current work): Search broken on beta cluster wikis - https://phabricator.wikimedia.org/T275975 (10EBernhardson) a:03EBernhardson [23:36:48] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Platonides) I don't think it's complicated at all. It should run fin on a ssh with a Match rule to only allow from extern... [23:37:39] 10Gerrit, 10GitLab (Initialization), 10User-brennen: Provide Speed & Function rough numbers for our current Gerrit web traffic - https://phabricator.wikimedia.org/T275564 (10Sergey.Trofimovsky.SF) Thank you, guys! That should be enough to start working on load tests, I'm sure we'll come up with more follow u... [23:43:17] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: forward external traffic to gitlab VMs (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) [23:43:50] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10Traffic, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: forward external traffic to gitlab VMs (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) [23:44:45] 10Release-Engineering-Team-TODO, 10DNS, 10Traffic, 10serviceops, and 2 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Reedy) [23:45:50] 10Release-Engineering-Team-TODO, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Dzahn) [23:45:55] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: forward external traffic to gitlab VMs (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) [23:46:44] 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: forward external traffic to gitlab VMs (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) This is partially blocked on T276170 .