[00:20:08] Project beta-update-databases-eqiad build #44032: 04FAILURE in 7.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/44032/ [00:25:54] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45 (10mmodell) Maybe we could do something as simple as showing a list of all the most recently submitted tasks on the submission pag... [00:25:58] sorry, I think I broke it [00:31:34] Amir1: broke what? :) [00:32:19] beta cluster update script [00:32:29] there's this red failure thingy right now [00:32:48] ah, ok [00:33:16] update = sync the puppetmaster? [00:34:13] presumably beta-update-databases-eqiad [00:34:52] the automatic database schema updater [01:21:59] Yippee, build fixed! [01:22:00] Project beta-update-databases-eqiad build #44033: 09FIXED in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/44033/ [01:24:51] \o/ [02:50:55] 10Continuous-Integration-Config: Add `ext-mysqli` to the composer-test CI container - https://phabricator.wikimedia.org/T226585 (10alex-mashin) Can automated tests read `extension.json`'s `requires.platform` section and add to the testing environment whatever is missing? [04:09:32] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 716 bytes in 0.508 second response time [04:09:32] PROBLEM - Host deployment-maps05 is DOWN: CRITICAL - Host Unreachable (172.16.4.168) [05:53:36] 10Release-Engineering-Team, 10Analytics-Radar, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10QChris) >>! In T230743#6363028, @mpopov wrote: > Requested `analytics/wmf-product/jobs` Gerrit repo Done. Cr... [06:19:11] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45 (10Aklapper) >>! In T45#6364677, @mmodell wrote: > Maybe we could do something as simple as showing a list of all the most recentl... [06:19:52] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45 (10Aklapper) Looking at the last 10000 tickets created, 4.19% of tickets marked as a duplicate. Might be biased (too recently crea... [06:48:12] 10Release-Engineering-Team-TODO, 10DBA, 10Product-Infrastructure-Team-Backlog: Drop DB tables for now-deleted zerowiki from production - https://phabricator.wikimedia.org/T227717 (10Marostegui) a:03Marostegui [08:22:43] 10Gerrit, 10Release-Engineering-Team (Development services), 10Repository-Admins: Decide whether requests for new Gerrit repos are handled on mediawiki.org or in Phabricator (currently it's both) - https://phabricator.wikimedia.org/T259147 (10hashar) [08:27:01] 10Gerrit, 10Release-Engineering-Team (Development services), 10Repository-Admins: Decide whether requests for new Gerrit repos are handled on mediawiki.org or in Phabricator (currently it's both) - https://phabricator.wikimedia.org/T259147 (10Majavah) [08:33:05] 10Release-Engineering-Team-TODO, 10DBA, 10Product-Infrastructure-Team-Backlog: Drop DB tables for now-deleted zerowiki from production - https://phabricator.wikimedia.org/T227717 (10Marostegui) 05Open→03Resolved All tables truncated: ` abuse_filter +----------+ | count(*) | +----------+ | 0 | +---... [08:33:12] 10Release-Engineering-Team-TODO, 10Product-Infrastructure-Team-Backlog, 10Wikimedia-Site-requests, 10Epic, and 2 others: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 (10Marostegui) [08:33:16] 10Gerrit, 10Release-Engineering-Team (Development services), 10Repository-Admins: Decide whether requests for new Gerrit repos are handled on mediawiki.org or in Phabricator (currently it's both) - https://phabricator.wikimedia.org/T259147 (10hashar) I have subscribed @QChris since he has been handling a lot... [08:37:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Audit stretch-backports usage in CI images - https://phabricator.wikimedia.org/T257553 (10hashar) From P11925: ` docker... [09:48:38] 10Phabricator: Phabricator: "500 Internal Server Error" when using VPN (due to specific IP range) - https://phabricator.wikimedia.org/T258059 (10NicoV) >>! In T258059#6362814, @Aklapper wrote: > @NicoV: Is this still a problem? Asking as the other ticket is resolved. @Aklapper I will be able to test if it works... [10:10:29] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [wikimedia-ch/wmch-infrastructure] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618725 [10:10:31] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [wikimedia-ch/wmch-infrastructure] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618725 (owner: 10QChris) [10:10:56] (03PS1) 10QChris: Import done. Revoke import grants [wikimedia-ch/wmch-infrastructure] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618726 [10:10:58] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [wikimedia-ch/wmch-infrastructure] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618726 (owner: 10QChris) [11:18:45] 10Gerrit: Gerrit shows error popup "Error 409 (Conflict): change is merged" - https://phabricator.wikimedia.org/T259790 (10Tgr) [11:21:39] 10Gerrit: Gerrit shows error popup "Error 409 (Conflict): change is merged" - https://phabricator.wikimedia.org/T259790 (10Tgr) Here's a full screenshot: {F31967650} Note how the changeset status pillbox in the top left corner is missing - normally before the commit ID and title it would say "Active" or "Merged"... [11:24:46] 10MediaWiki-Codesniffer, 10Patch-For-Review: Add sniff to disallow new global variable imports in functions (using global keyword) - https://phabricator.wikimedia.org/T257710 (10hashar) [11:50:38] 10Phabricator: Imported bugzilla comment by specific account (mwjames) has "Unknown Object (User)" as commenter (does not happen as task author) - https://phabricator.wikimedia.org/T85203 (10Aklapper) 05Open→03Declined Right, thanks for checking. Boldly declining. [12:02:55] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10Lucas_Werkmeister_WMDE) [13:30:55] 10Release-Engineering-Team-TODO, 10Product-Infrastructure-Team-Backlog, 10Wikimedia-Site-requests, 10Epic, and 2 others: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 (10MaxSem) [13:32:05] 10Release-Engineering-Team-TODO, 10Product-Infrastructure-Team-Backlog, 10Wikimedia-Site-requests, 10Epic, and 2 others: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 (10MaxSem) 05Open→03Resolved \m/ [13:59:36] 10Release-Engineering-Team, 10Analytics-Radar, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) [13:59:50] 10Release-Engineering-Team, 10Analytics-Radar, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) [14:00:21] 10Release-Engineering-Team, 10Analytics-Radar, 10Product-Analytics, 10Repository-Admins: Create a repository and user for Product Analytics Oozie jobs - https://phabricator.wikimedia.org/T230743 (10mpopov) 05Open→03Resolved Thank you @QChris! [14:09:23] (03PS1) 10Hashar: Add debian-glue to operations/software/transferpy [integration/config] - 10https://gerrit.wikimedia.org/r/618756 (https://phabricator.wikimedia.org/T253736) [14:10:24] (03CR) 10Hashar: "That would make CI to attempt to build the package when a patchset is proposed. Non voting for now cause I have no idea whether it will pa" [integration/config] - 10https://gerrit.wikimedia.org/r/618756 (https://phabricator.wikimedia.org/T253736) (owner: 10Hashar) [14:10:29] (03CR) 10jerkins-bot: [V: 04-1] Add debian-glue to operations/software/transferpy [integration/config] - 10https://gerrit.wikimedia.org/r/618756 (https://phabricator.wikimedia.org/T253736) (owner: 10Hashar) [14:14:57] (03PS2) 10Hashar: Add debian-glue to operations/software/transferpy [integration/config] - 10https://gerrit.wikimedia.org/r/618756 (https://phabricator.wikimedia.org/T253736) [14:20:43] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are half the speed of newer ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) 05Open→03Declined That most probably comes from the CPU scaling BIOS setting described at T225713... [14:22:38] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (ARCHIVED -- Shared Build Failure): Occasional Selenium build failures related to temporary directories / no space left - https://phabricator.wikimedia.org/T243223 (10hashar) No idea what happened in January 2020. I guess some cache got over... [14:31:28] (03PS1) 10Hashar: jjb: remove no more used castor-load macro [integration/config] - 10https://gerrit.wikimedia.org/r/618760 [14:31:49] (03CR) 10Hashar: "That should be a noop in the JJB configuration ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/618760 (owner: 10Hashar) [14:44:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Assess whether we should still disable seccomp in Docker for CI - https://phabricator.wikimedia.org/T249729 (10hashar) 05Open→03Resolved [14:50:34] (03CR) 10Ahmon Dancy: [C: 03+2] jjb: remove no more used castor-load macro [integration/config] - 10https://gerrit.wikimedia.org/r/618760 (owner: 10Hashar) [14:51:40] (03Merged) 10jenkins-bot: jjb: remove no more used castor-load macro [integration/config] - 10https://gerrit.wikimedia.org/r/618760 (owner: 10Hashar) [14:52:38] What I ended up doing to get my blubber/pipelinelib running with all my preporatory steps, was to use builder/require [ . ]. That copies all the files for me. Then I had to make the script the command execute to move all the files to a new subdirectory of /srv (since it lives in /srv to gain access to add other /srv/subdirs from that script). Thus I also removed all my copies-statements from the blubber yaml. How crazy is this? [15:24:40] 10Continuous-Integration-Infrastructure, 10Operations, 10Jenkins, 10Patch-For-Review: Update Jenkins gpg release key in reprepro - https://phabricator.wikimedia.org/T259116 (10akosiaris) > I could not find where we store that key in puppet :-\ That's cause we don't store it. We just use the fingerprint. [15:29:46] !log Restarted puppetdb on deployment-puppetdb03.deployment-prep.eqiad.wmflabs. Dead from OOM. [15:29:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:01:02] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): echostore helm test service checker failing in staging cluster - https://phabricator.wikimedia.org/T259686 (10akosiaris) That not very helpful message was due to pretty old image of service-checker in the image. I 'v... [16:05:03] 10Beta-Cluster-Infrastructure, 10VPS-Projects, 10Puppet: Puppet failures on deployment-docker-changeprop01, deployment-docker-cpjobqueue01, deployment-push-notifications01, and deployment-docker-proton01 due to Docker version pinning - https://phabricator.wikimedia.org/T259812 (10bd808) [16:37:47] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10MediaWiki Train Development Environment: train-dev: actually install pubkeys/* into the cloud-init files - https://phabricator.wikimedia.org/T259610 (10thcipriani) 05Open→03Invalid We ended up using a differe... [16:42:18] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10MediaWiki Train Development Environment: train-dev: Use fixed MAC addresses for the VMs - https://phabricator.wikimedia.org/T259609 (10dancy) Note that this means that you could (or should) not run two of these V... [16:49:46] (03PS1) 10Jforrester: layout: [mediawiki/extensions/WikiLambda] Add extension-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/618789 [16:49:58] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/extensions/WikiLambda] Add extension-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/618789 (owner: 10Jforrester) [16:51:29] (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/WikiLambda] Add extension-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/618789 (owner: 10Jforrester) [16:53:31] !log Zuul: [mediawiki/extensions/WikiLambda] Add extension-coverage [16:53:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:05:18] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Sketch MediaWiki production image composition and workflows - https://phabricator.wikimedia.org/T259817 (10dduvall) [17:05:43] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Sketch MediaWiki production image composition and workflows - https://phabricator.wikimedia.org/T259817 (10dduvall) [17:05:45] 10Release-Engineering-Team-TODO, 10Release Pipeline, 10serviceops-radar, 10Epic: Get production MW-land images built and published - https://phabricator.wikimedia.org/T238771 (10dduvall) [17:14:55] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Sketch MediaWiki production image composition and workflows - https://phabricator.wikimedia.org/T259817 (10dduvall) p:05Triage→03Medium a:03dduvall [17:28:54] 10Beta-Cluster-Infrastructure, 10VPS-Projects, 10Puppet: Puppet failures on deployment-docker-changeprop01, deployment-docker-cpjobqueue01, deployment-push-notifications01, deployment-docker-mobileapps01, and deployment-docker-proton01 due to Docker version pinning - https://phabricator.wikimedia.org/T259812 (... [17:40:23] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10brennen) [17:40:30] 10Beta-Cluster-Infrastructure, 10VPS-Projects, 10Patch-For-Review, 10Puppet, and 2 others: Puppet failures on deployment-docker-changeprop01, deployment-docker-cpjobqueue01, deployment-push-notifications01, deployment-docker-mobileapps01, and deployment-docker-prot... - https://phabricator.wikimedia.org/T259812 [17:40:57] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10brennen) Rolling forward to group1 again. [17:43:45] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 302 Found - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 715 bytes in 0.069 second response time [18:07:43] (03CR) 10Ahmon Dancy: [C: 04-1] "I propose abandoning this commit. I don't think it will have much impact, especially after the work that Antoine did to reduce workspace " [integration/config] - 10https://gerrit.wikimedia.org/r/614879 (https://phabricator.wikimedia.org/T258448) (owner: 10Brennen Bearnes) [18:08:46] (03CR) 10Brennen Bearnes: "Reasonable. We can always revisit if it turns out to be useful." [integration/config] - 10https://gerrit.wikimedia.org/r/614879 (https://phabricator.wikimedia.org/T258448) (owner: 10Brennen Bearnes) [18:08:56] (03Abandoned) 10Brennen Bearnes: WIP: maintenance-disconnect-full-disks: remove old workspaces [integration/config] - 10https://gerrit.wikimedia.org/r/614879 (https://phabricator.wikimedia.org/T258448) (owner: 10Brennen Bearnes) [18:22:02] 10Beta-Cluster-Infrastructure, 10VPS-Projects, 10Puppet: Puppet failures on deployment-docker-changeprop01, deployment-docker-cpjobqueue01, deployment-push-notifications01, deployment-docker-mobileapps01, and deployment-docker-proton01 due to Docker version pinning - https://phabricator.wikimedia.org/T259812 (... [18:28:11] 10Release-Engineering-Team, 10Vue.js (Vue.js-Search): Work with RelEng to add PipelineBot to Vector - https://phabricator.wikimedia.org/T257582 (10dduvall) >>! In T257582#6364175, @Niedzielski wrote: > I'm worried there's a subtlety to your understanding of Gerrit post-merge and my coarse understanding of Git... [18:45:37] (03PS1) 10Ottomata: Add wikimedia-event-utilities project and CI jobs [integration/config] - 10https://gerrit.wikimedia.org/r/618801 (https://phabricator.wikimedia.org/T251609) [18:51:57] (03CR) 10Ottomata: "Not sure if this is correct, but I'd like to be able to trigger release builds for this repository much like we do for analytics/refinery/" [integration/config] - 10https://gerrit.wikimedia.org/r/618801 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [19:38:24] James_F, etc: 1.36.0-wmf.3 seems not to have the latest code from the 1.36.0-wmf.3 branch of mediawiki-vendor [19:38:31] (in particular, parsoid -a4) [19:39:03] how did that happen? why didn't https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/618038 (merged monday) work? [19:39:19] is this our bug, or releng's? [19:43:27] 10Release-Engineering-Team, 10Parsoid: Deployed 1.36.0-wmf.3 does not have the 1.36.0-wmf.3 branch of mediawiki-vendor - https://phabricator.wikimedia.org/T259832 (10cscott) [19:43:56] 10Release-Engineering-Team, 10Parsoid: Deployed 1.36.0-wmf.3 does not have the 1.36.0-wmf.3 branch of mediawiki-vendor - https://phabricator.wikimedia.org/T259832 (10cscott) [19:43:59] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10cscott) [19:44:08] cscott: unclear on this one [19:45:07] 10Release-Engineering-Team, 10Parsoid: Deployed 1.36.0-wmf.3 does not have the 1.36.0-wmf.3 branch of mediawiki-vendor - https://phabricator.wikimedia.org/T259832 (10cscott) (I'm adding this to the blocker task for 1.36.0-wmf.3, but to be clear we don't actually consider this a blocker; there's no show-stopper... [19:45:11] cscott: did that merge after branch cut? [19:45:38] brennen: yes. [19:46:29] i don't know exactly when the branch cut is (it's automatic, isn't it?) but it was w/in a couple of hours of it, I think. certainly well before the group0 train started to roll. [19:46:49] so i think that probably what needs to happen is for vendor to get updated on the deploy box and synced [19:46:59] 10Release-Engineering-Team, 10Parsoid: Deployed 1.36.0-wmf.3 does not have the 1.36.0-wmf.3 branch of mediawiki-vendor - https://phabricator.wikimedia.org/T259832 (10Legoktm) For some reason Gerrit didn't automatically create a submodule bump in mediawiki/core... https://gerrit.wikimedia.org/r/plugins/gitiles/... [19:47:28] legoktm seems to have some thoughts? ^ [19:47:34] i haven't previously dealt specifically with backports to vendor so i'm not sure if i'm missing something [19:47:56] normally the submodule bumps to core are supposed to be automatic [19:48:04] yeah, i was just wondering about that [19:48:07] I'm not sure why it didn't create one for vendor [19:48:16] at any rate, we don't need to fix it for this train, i'd like like to figure out what the right process is (which might just mean "find the train conductor and poke them") so we can write it into our deploy procedures [19:48:28] or fix the automatic bump, i guess, if it's a bug [19:48:30] it's certainly working for extensions/skins, wasn't sure if it was in place for vendor as well [19:50:22] i wonder if we should actually revert this on the wmf.3 vendor branch [19:50:28] so that git matches what is actually deployed [19:50:35] either that or do the submodule bump manually and sync it out [19:51:17] how do rollbacks work? if the wmf.4 train gets rolled back, is there any chance that the actual head of the vendor branch might get deployed? [19:52:15] i don't particularly care whether we end up on parsoid -a3 or -a4 for wmf.3, but i'd like to avoid the case where we shift unexpectedly during a rollback of wmf.4 or something like that [19:52:24] cscott: rollbacks work by reverting the commit with changes to wikiversions.json. [19:52:28] deploy state comes from the submodules [19:52:44] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 36448 bytes in 1.259 second response time [19:53:12] so in some sense we're still deployed in a reproducible config; aka the deployed state matches the submodule hash in core [19:53:24] correct. [19:53:37] ok, well that's reassuring at least. [19:53:44] the potential problem comes when someone else wants to backport something in vendor and stumbles into undeployed stuff [19:53:50] i think in this case we should probably bump the submodule manually and sync it out, and more generally we should figure out whether something needs to be changed for the vendor submodule. [19:54:13] (i should know how the automatic submodule bump happens, but at the moment i don't.) [19:55:47] https://gerrit-review.googlesource.com/Documentation/user-submodules.html [19:56:28] ah, nice [19:56:28] (03PS1) 10Legoktm: Edit Repo Config [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618808 [19:56:57] found it [19:57:03] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 24660 bytes in 1.158 second response time [19:58:37] (03PS2) 10Legoktm: Enable automatic submodule bumps for mediawiki/vendor [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618808 (https://phabricator.wikimedia.org/T259832) [20:01:00] I'm not sure that's exactly right [20:02:53] legoktm: yeah, we can probably get a couple of folks to review. [20:04:45] because vendor isn't the superproject [20:04:47] it's the subproject [20:05:48] (03CR) 10Legoktm: "I think this is backwards in that mediawiki/vendor is the subproject, not the superproject? In any case, my guess is that some config rela" [core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618808 (https://phabricator.wikimedia.org/T259832) (owner: 10Legoktm) [20:08:04] meantime, shall i go ahead and fetch latest version of wmf/1.36.0-wmf.3 on vendor/, add to parent, push both, and sync? [20:08:21] (er, push, not "push both".) [20:08:51] submodules: still make my head hurt. [20:11:08] 10Gerrit, 10Release-Engineering-Team (Development services), 10Repository-Admins: Decide whether requests for new Gerrit repos are handled on mediawiki.org or in Phabricator (currently it's both) - https://phabricator.wikimedia.org/T259147 (10QChris) >>! In T259147#6365032, @hashar wrote: > I have at least a... [20:14:59] brennen, works for me. /cc cscott [20:18:25] ack, doing. [20:19:48] yes. i guess for the time being i'll put something about 'if cherry-picking to vendor, check the submodules in core are updated' into our deploy process notes [20:20:20] i thought for sure we'd cherry-picked to vendor before, but maybe James_F or whoever was driving the cherry pick manually did the core submodules update for us [20:21:10] i guess this feels like something that just hasn't come up super often [20:21:59] well, since parsoid is deployed via vendor now, any train-related parsoid patch needs to go through vendor. so it might come up more often from now on. [20:22:13] yep, good to be getting it figured out. [20:22:33] prior to this train i hadn't been aware that parsoid deploys were via vendor. [20:23:09] it's our great sacrifice to the train gods. we used to have our own totally independent build process to our own totally independent servers. [20:23:24] but we are being swallowed by the uniform prod gods ;) [20:25:10] ('release process' not 'build process'. and i don't really miss git-deploy at *all*) [20:39:08] brennen: shouldn't I see a commit to https://gerrit.wikimedia.org/g/mediawiki/core/+/refs/heads/wmf/1.36.0-wmf.3 with your vendor bump? [20:39:26] cscott: sorry, still sorting this out. [20:39:36] ok, just checking that i understand the process [20:40:03] realized i need to create that commit without also pushing any security patches to the public wmf/1.36.0-wmf.3. [20:42:49] 10Beta-Cluster-Infrastructure, 10The-Wikipedia-Library: Email spam from Betawiki - https://phabricator.wikimedia.org/T256297 (10XanonymusX) Another friendly reminder: I'm still drowning in WP Library emails every day. Please do something about it! [20:49:32] cscott: there's that commit - https://gerrit.wikimedia.org/r/c/mediawiki/core/+/618850 - guess i'll wait for gate-and-submit, rebase, and sync. [20:51:27] brennen: cool. i also uploaded one to gerrit which you can ignore, i was just testing the docs I was writing. [20:52:45] https://wikitech.wikimedia.org/w/index.php?title=Parsoid&type=revision&diff=1876779&oldid=1876164 [21:46:55] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Parsoid, 10Patch-For-Review, 10User-brennen: Deployed 1.36.0-wmf.3 does not have the 1.36.0-wmf.3 branch of mediaw... - https://phabricator.wikimedia.org/T259832 [21:47:41] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Parsoid, 10Patch-For-Review, 10User-brennen: mediawiki-vendor submodule doesn't get automatically bumped on releas... - https://phabricator.wikimedia.org/T259832 [22:03:01] PROBLEM - Host deployment-chromium01 is DOWN: CRITICAL - Host Unreachable (172.16.4.108) [22:42:25] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): echostore helm test service checker failing in staging cluster - https://phabricator.wikimedia.org/T259686 (10jeena) Thanks for taking a look. I should have confirmed the service-checker version! [23:03:53] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10brennen) [23:03:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Parsoid, 10Patch-For-Review, 10User-brennen: mediawiki-vendor submodule doesn't get automatically bumped on releas... - https://phabricator.wikimedia.org/T259832 [23:04:40] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T257971 (10brennen) Just filed: T259856 As mentioned there, I'm reluct... [23:46:41] !log Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/601429 in beta. [23:46:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL