[02:11:57] (03PS7) 1020after4: Fix up branch.py so that it's suitable for wmf/ production branches [tools/release] - 10https://gerrit.wikimedia.org/r/543248 [05:25:44] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10Marostegui) 05Resolved→03Open After the upgrade (T222391) the same behaviour as reported on this original ticket has showed up again with: https://gerrit.wikimedia.org/r/#/c/operat... [07:17:48] (03CR) 10Awight: "Thanks!" [tools/release] - 10https://gerrit.wikimedia.org/r/544899 (owner: 10Thcipriani) [07:33:17] 10Gerrit: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Marostegui) @Paladox pointed me to this ticket since I reopened T234533#5593695 earlier today. Feel free to close that one and follow up on this task Thanks! [07:46:04] 10Gerrit: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) More changes have been reported missing T234533#5593695 * https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/540006/ Has been merged, it is in cobalt and show has merged/reveted etc but on gerrit1001 t... [07:49:50] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10awight) Same here, I found this merge/unmerged patch: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Popups/+/538641/ Worryingly, there's no record of the patch every having... [07:50:06] 10Gerrit, 10Release-Engineering-Team-TODO: gerrit shows merged patch as pending - https://phabricator.wikimedia.org/T234533 (10hashar) 05Open→03Resolved >>! In T234533#5593695, @Marostegui wrote: > After the upgrade (T222391) the same behaviour as reported on this original ticket has showed up again with:... [07:51:17] 10Gerrit: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10awight) Here's another corrupted patch, merged but Gerrit displays as unmerged: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Popups/+/538641/ [08:03:53] 10MediaWiki-Codesniffer, 10LibUp: Upgrade PHPCS to 28.0.0 in all repos - https://phabricator.wikimedia.org/T235113 (10Daimona) >>! In T235113#5592902, @Legoktm wrote: >>>! In T235113#5569899, @Daimona wrote: >> And also, please upgrade wikibase/wikibase-codesniffer to 1.0.0, which brings in the 28.0.0 version... [08:20:12] (03CR) 10Hashar: [C: 03+2] "Yeah I guess that is bad copy pasta :]" [integration/config] - 10https://gerrit.wikimedia.org/r/544948 (owner: 10Hashar) [08:23:03] (03Merged) 10jenkins-bot: jjb: archive section is invalid, use "publishers" instead [integration/config] - 10https://gerrit.wikimedia.org/r/544948 (owner: 10Hashar) [08:30:51] (03CR) 10Hashar: [C: 03+2] Add missing docker-cleanup macros [integration/config] - 10https://gerrit.wikimedia.org/r/544952 (owner: 10Hashar) [08:33:47] (03Merged) 10jenkins-bot: Add missing docker-cleanup macros [integration/config] - 10https://gerrit.wikimedia.org/r/544952 (owner: 10Hashar) [08:51:33] (03CR) 10Hashar: [C: 04-1] Upgrade PostBuildScript and Update jjb to 2.0.3 (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [08:52:00] (03PS4) 10Hashar: Upgrade PostBuildScript and Update jjb to 2.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) [09:06:14] (03CR) 10Jforrester: Upgrade PostBuildScript and Update jjb to 2.0.3 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [09:12:56] (03PS3) 10Jforrester: jjb: use Fresnel 0.5.0 [integration/config] - 10https://gerrit.wikimedia.org/r/545005 (https://phabricator.wikimedia.org/T235195) (owner: 10Phedenskog) [09:25:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (201910), 10PHP 7.4 support: Provide CI testing for PHP 7.4 - https://phabricator.wikimedia.org/T236132 (10Jdforrester-WMF) [09:39:02] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/tools/minus-x] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544813 (owner: 10Jforrester) [09:39:13] (03PS2) 10Jforrester: layout: [mediawiki/tools/phpunit-patch-coverage] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544814 [09:39:20] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/tools/phpunit-patch-coverage] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544814 (owner: 10Jforrester) [09:39:28] (03PS2) 10Jforrester: layout: [wikimedia/lucene-explain-parser] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544815 [09:39:33] (03CR) 10Jforrester: [C: 03+2] layout: [wikimedia/lucene-explain-parser] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544815 (owner: 10Jforrester) [09:39:41] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/services/jobrunner] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544816 (owner: 10Jforrester) [09:39:47] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/oauthclient-php] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544817 (owner: 10Jforrester) [09:39:54] (03CR) 10Jforrester: [C: 03+2] layout: [cdb] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544818 (owner: 10Jforrester) [09:40:31] (03Merged) 10jenkins-bot: layout: [mediawiki/tools/minus-x] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544813 (owner: 10Jforrester) [09:41:11] (03Merged) 10jenkins-bot: layout: [mediawiki/tools/phpunit-patch-coverage] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544814 (owner: 10Jforrester) [09:41:15] (03Merged) 10jenkins-bot: layout: [wikimedia/lucene-explain-parser] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544815 (owner: 10Jforrester) [09:45:03] (03CR) 10Hashar: Upgrade PostBuildScript and Update jjb to 2.0.3 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [09:50:19] (03PS5) 10Hashar: Upgrade PostBuildScript and Update jjb to 2.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) [10:01:25] (03PS6) 10Hashar: Upgrade PostBuildScript and Update jjb to 2.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) [10:03:10] (03CR) 10Hashar: "PS5 fixed a few more :)" (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [10:04:27] hmm [10:04:40] James_F: I think I am going to try the postbuild script upgrade after lunch [10:04:45] and hope for the best ;] [10:09:36] (03CR) 10Jforrester: [C: 03+1] Upgrade PostBuildScript and Update jjb to 2.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [10:27:15] (03PS2) 10Jforrester: layout: [mediawiki/services/jobrunner] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544816 [10:27:20] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/544816 (owner: 10Jforrester) [10:27:29] (03PS2) 10Jforrester: layout: [mediawiki/oauthclient-php] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544817 [10:27:40] (03PS2) 10Jforrester: layout: [cdb] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544818 [10:27:50] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/544818 (owner: 10Jforrester) [10:27:57] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/544817 (owner: 10Jforrester) [10:28:49] (03Merged) 10jenkins-bot: layout: [mediawiki/services/jobrunner] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544816 (owner: 10Jforrester) [10:29:35] (03Merged) 10jenkins-bot: layout: [mediawiki/oauthclient-php] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544817 (owner: 10Jforrester) [10:29:39] (03Merged) 10jenkins-bot: layout: [cdb] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544818 (owner: 10Jforrester) [10:34:21] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 3 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10fgiunchedi) FTR: as of indices starting on Oct 22nd the limit is now 20... [10:43:33] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 3 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10hashar) >>! In T234564#5592805, @matmarex wrote: > I tested that on mwd... [10:45:50] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 3 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10matmarex) Oh, thanks! Just in time for the SWAT, let's try… [10:50:57] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 4 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10fgiunchedi) >>! In T234564#5592805, @matmarex wrote: > I tested that on... [10:54:09] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 4 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10matmarex) >>! In T234564#5594395, @fgiunchedi wrote: > Did the logs wit... [10:56:33] 10MediaWiki-Codesniffer, 10LibUp: Upgrade PHPCS to 28.0.0 in all repos - https://phabricator.wikimedia.org/T235113 (10Jdforrester-WMF) [10:58:29] 10Gerrit: Already merged code-reviews show up on gerrit - https://phabricator.wikimedia.org/T236135 (10MarcoAurelio) [11:28:28] 10Gerrit, 10Release-Engineering-Team-TODO: Already merged code-reviews show up on gerrit - https://phabricator.wikimedia.org/T236135 (10Jdforrester-WMF) p:05Triage→03Unbreak! [11:30:44] 10Gerrit, 10Release-Engineering-Team-TODO: Already merged code-reviews show up on gerrit - https://phabricator.wikimedia.org/T236135 (10Marostegui) That's already reported: T236114 [11:30:47] 10Gerrit, 10Release-Engineering-Team-TODO: Already merged code-reviews show up on gerrit - https://phabricator.wikimedia.org/T236135 (10Tchanders) I'm seeing the same thing in MediaWiki core - several of my merged patches are showing up as open in my outgoing reviews queue. Most obvious example is: https://ge... [11:31:10] 10Gerrit: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Marostegui) More reports at: T236135 [11:31:55] 10Gerrit: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Marostegui) [11:31:57] 10Gerrit, 10Release-Engineering-Team-TODO: Already merged code-reviews show up on gerrit - https://phabricator.wikimedia.org/T236135 (10Marostegui) [11:32:24] 10Gerrit: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Marostegui) p:05Normal→03High [11:32:56] 10Gerrit, 10Release-Engineering-Team-TODO: Already merged code-reviews show up on gerrit - https://phabricator.wikimedia.org/T236135 (10Tchanders) Thanks @Marostegui [11:33:01] 10Gerrit: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Jdforrester-WMF) [11:33:03] 10Gerrit, 10Release-Engineering-Team-TODO: Already merged code-reviews show up on gerrit - https://phabricator.wikimedia.org/T236135 (10Jdforrester-WMF) [11:33:47] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Jdforrester-WMF) UBN? [11:40:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10hashar) p:05Triage→03High Summary: the Xdebug package for `stretch-wikimedia` `component/p... [11:42:17] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 4 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10matmarex) It works now! Example: https://logstash.wikimedia.org/goto/a8... [11:45:50] 10Gerrit, 10Operations: Editing in Gerrit isn't saved after the update/migration to gerrit1001 - https://phabricator.wikimedia.org/T236143 (10MoritzMuehlenhoff) [11:47:18] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10MoritzMuehlenhoff) Nothing critical, but this happens after the updat... [11:51:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10jijiki) @hashar I will look into it, sorry for not getting to it sooner [12:01:26] 10Continuous-Integration-Config, 10MediaViewer, 10MediaWiki-Core-Testing, 10MobileFrontend, and 7 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10awight) [12:02:05] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10Move-Files-To-Commons, 10TCB-Team, and 5 others: Test WikiTextEditor class with browser tests. - https://phabricator.wikimedia.org/T190829 (10awight) [12:02:08] 10Continuous-Integration-Infrastructure: Need a mechanism for injecting a LocalSettings.php fragment before nodejs browser tests - https://phabricator.wikimedia.org/T223431 (10awight) [12:02:14] 10Continuous-Integration-Config, 10MediaViewer, 10MediaWiki-Core-Testing, 10MobileFrontend, and 7 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10awight) [12:02:15] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Quibble, and 2 others: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10awight) [12:10:05] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) So for https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/540006/ , I looked at the actual git repositories on the server. The repository is in `/srv/gerrit/git/opera... [12:10:32] so gerrit [12:10:37] and poking paladox about it :) [12:10:54] what I found for ONE of the change that does not show up as up-to-date on gerrit1001 is [12:11:01] * zeljkof is still holding breath [12:11:10] that there is a file for refs/changes/06/540006/meta [12:11:20] which is from October 1st 4:55 [12:11:24] that points to some old commit [12:11:39] hashar, I'm running branch cut now, so be mindful ot that if you fix things [12:11:40] and is reflecting on https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/540006/ [12:11:42] because well [12:11:48] the ui does use that meta file [12:12:30] Yeh [12:12:42] but that meta file points to an obsolete state [12:12:47] the state of that change on october 1st [12:12:50] meanwhile on cobalt [12:13:02] We had that issue with some of mutante changes and mine [12:13:07] the reference as further advanced since that change got commented on, merged and even a revert has been done [12:13:12] Tyler wrote the command he used in -operations [12:13:15] so it is more up to date on cobalt [12:13:58] but on cobalt that file refs/changes/06/540006/meta  does not exist [12:14:29] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) And obviously on cobalt the file `refs/changes/06/540006/meta` does not exist. It has been packed in `.git/packed-refs`. [12:14:32] the reference is in .git/packed-refs [12:14:37] and has been updated on cobalt [12:14:59] but the refs/changes/06/540006/meta got copied to gerrit1001 at some point [12:15:09] I guess a rsync has been done ahead of time which carried those files [12:15:28] and the final rsync didn't use --delete? [12:15:38] so I suspect that during the migration, when doing the final rsync from cobalt to gerrit1001, --delete has NOT been used [12:15:47] and some meta file got stall and stick on gerrit1001 [12:15:53] thus not reflecting the current state of cobalt [12:15:54] but [12:16:00] the packed-refs got synced [12:16:04] so the proper metadata are there [12:16:24] liw: yes [12:16:26] well [12:16:32] no I can not affirm that [12:16:39] but that sounds like a plausible explanation [12:16:58] so we have gerrit1001 still carrying some obsolete state in the form of some files under the .git directories [12:16:59] who did the final rsync? can we get them to check their shell history? [12:17:00] Yeh we didn’t use —delete [12:19:20] hashar's analysis sounds plausible; I hope it ends up on the relevant ticket :) [12:19:40] rsync is annoyingly difficult to use correctly, too many gotchas [12:20:13] a good takeaway from this might be to write a script around rsync that gives the myriad of options to make a correct copy including all corner cases [12:20:22] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10Move-Files-To-Commons, 10TCB-Team, and 5 others: Test WikiTextEditor class with browser tests. - https://phabricator.wikimedia.org/T190829 (10awight) 05Open→03Resolved [12:20:30] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) The devil is that we might have a split brain if a change with a `meta` file on gerrit1001 has been updated when it also has been updated on cobalt after the inital sync :-\ But I g... [12:20:48] paladox: yeah so that has left over a bunch of obsolete files on gerrit1001 :-\ [12:21:51] hashar, is there a way we can detect the stale ones? [12:22:12] :( [12:22:19] liw: thcipriani had a script to find them. [12:22:36] I dont know [12:22:46] gotta write an analysis / decision tree ;D [12:23:14] and have to figure out how to find whether the state is in packed-refs and wether it is accurate [12:23:35] I am not sure what happens if one comment on a change that has an obsolete meta file [12:26:31] 10Deployments, 10Release-Engineering-Team, 10serviceops, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10CDanis) This happened yesterday, during firefighting work on {T235949} All appservers had the same config: {P9416}... [12:36:39] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Wikidata-Campsite, 10Patch-For-Review: [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10Addshore) @hashar do you think we can try re adding this extension to the gate jobs and see h... [12:37:21] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10Move-Files-To-Commons, 10TCB-Team, and 5 others: Test WikiTextEditor class with browser tests. - https://phabricator.wikimedia.org/T190829 (10awight) 05Resolved→03Open Oops, the main patch is still pen... [12:41:08] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) One can list all the meta files and their sha1 using: ` (for x in $(find . -path '*/refs/changes/*/*/meta'); do grep -H . $x; done;)|sort ` And diffing gives a bunch of various diff... [12:52:12] (03PS1) 10Jforrester: dockerfiles: Provide initial PHP74 images (based on 7.4.0-rc.3 for now) [integration/config] - 10https://gerrit.wikimedia.org/r/545269 (https://phabricator.wikimedia.org/T236132) [12:54:38] hashar: oh! that makes total sense! [12:55:36] hashar: would the test for that be, moving the loose ref file for that change to a temporary location and seeing what happens? [12:55:47] I dont know really [12:55:56] gotta think about all the various implication [12:56:04] I guess we can parse the current state [12:56:36] I guess my question is: if we have both a ref in a packfile and the same ref loose on disk: which does gerrit think is real? [12:56:46] the loose ref [12:56:49] well [12:56:54] right, if your hunch is correct [12:56:55] at least for the change I experimented [12:56:56] with [12:57:03] which would match gerrit 2.11.0 implementation [12:57:11] err [12:57:15] probably not a lot of reason for that to change [12:57:19] which match git 2.11.0 implementation [12:57:32] it first look up for a loose reference and if the file is not found, check packed-refs [12:57:51] seems like the most efficient way to do it [12:58:11] I'm sure this has been considered, but should we put Gerrit into read-only mode until this is resolved? [13:01:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10MoritzMuehlenhoff) Where does this error show up, in some CI instance or in prod? The only pro... [13:01:42] that would mean my branch cutting and possibly train deployment will be delaed [13:01:59] !log Limited pywikibot to one concurrent job in Wikimedia TravisCI to avoid resource exhaustion [13:02:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:11:02] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) >>! In T236114#5594704, @hashar wrote: > My suspicion is that after Gerrit got stopped on cobalt, a rsync has been done that DID NOT DELETE FILES. Thus we carried over a lot of m... [13:13:21] I think I got a differ [13:17:35] while waiting for branch cut, wrote this: http://paste.debian.net/1109123/ [13:18:59] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) I went with a terrible script: {P9435} Output: {P9436} Which is not that helpful to analysis :\ [13:19:58] alias cp="rsync -avp" [13:19:58] :D [13:20:16] Project beta-scap-eqiad build #272277: 04FAILURE in 5 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272277/ [13:23:25] so I guess my trouble [13:23:40] is to identify which meta files turn out to be problematic :-\ [13:24:29] Project beta-scap-eqiad build #272278: 04STILL FAILING in 7.6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272278/ [13:24:51] hashar: I think I have a solution [13:25:01] while read ref; do fn="$(echo "$ref" | awk '/meta/ {print $2}')"; if [ "$fn" != '' ]; then stat -c %n "$fn" 2>&1; fi; done < packed-refs | grep -v 'No such file' [13:25:38] ^ read packed-refs, if the ref file also exists loose on disk: it's a problem [13:26:05] running that inside an individual repo shows all the meta refs that differ [13:27:41] hashar: I think I want to: stop gerrit, backup repos, rm all refs from disk that exist both on disk and in packed-refs; done. Thoughts on that plan? [13:28:14] ^ or paladox [13:28:20] ^ or mutante [13:29:05] that depends whether the ref got changed ? [13:29:43] imagine on cobalt that we have rsynced a meta file pointing to A [13:29:50] no, it just looks to see if the ref was packed [13:29:55] err [13:29:56] oh [13:30:13] In case this is helpful, here's an example where I made comments on a patch after discovering the corruption: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Popups/+/538641/ -- if the CR-2 disappears, then we're losing today's data. [13:30:13] oh [13:30:20] so you "parse" the packed-refs file?! [13:30:25] right [13:31:10] on gerrit1001 all those meta file should be valid references in the pack file for sure [13:31:26] and some would point to an obsolete sha1 [13:31:36] the devil is figuring out whether there is a newer one in the packed-refs [13:32:32] trying to think about all the race conditions that might have occured :-\ [13:32:51] hashar: I would say there are going to be fewer of those "newer" ones and we can handle on a case-by-case? [13:33:02] yeah sure [13:33:12] but can we find a way to detect them ? [13:33:23] like awight 's -2 above. A lot of the newer ones are probably confusion and noise [13:33:25] hey just FYI weird thing: gerrit lost patchset 6 on this change I was just working on: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventLogging/+/524575/ [13:33:42] it doesn't matter for my case, I just pushed it again, but it's weird so thought yall should know [13:33:47] milimetric: we're just looking into that https://phabricator.wikimedia.org/T236114 [13:33:49] also, congrats thcipriani, saw your promotion [13:33:54] milimetric: thanks [13:34:20] ok, cool, good luck, let me know if you need any details about it, I pushed it Thursday night around 18:30 EST I think [13:34:53] https://docs.google.com/document/d/190jEWpgmviywSa8Q2DPE0xv79jyBTaV9fUcWQtTcwuY/edit [13:34:54] milimetric: lemme try something. [13:34:56] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10Move-Files-To-Commons, 10TCB-Team, and 5 others: Test WikiTextEditor class with browser tests. - https://phabricator.wikimedia.org/T190829 (10awight) 05Open→03Resolved [13:34:58] err [13:35:13] so my worry is that if one of the affected change got altered on gerrit1001 [13:35:24] I would guess that the meta file has been updated [13:35:27] Project beta-scap-eqiad build #272279: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272279/ [13:35:43] and maybe the change has been updated on cobalt previously and that would reflect in the packed-refs [13:35:47] so essentially we get a fork [13:40:55] hashar: so milimetric 's change is interesting. packed-refs is the same as the ref on disk. Patchset 6 is still missing. [13:41:40] hmm [13:42:22] milimetric: patchset 6 is back [13:42:32] your new patchset is gone [13:42:35] oh, interesting, happened to another change of mine: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/530878/ [13:42:51] milimetric: don't update that one yet please [13:42:51] well it is on cobalt at least :) [13:43:05] hashar: I did an update ref on that one [13:43:10] one fewer to fix later [13:43:46] wdiff -3 --avoid-wraps cobalt-meta-files.txt gerrit1001-meta-files.txt|colordiff -u [13:43:50] gives me some interesting infos [13:44:09] * milimetric backs away slowly and lets the professionals fight the scary machine [13:44:18] which I cant gasp ... :\ [13:44:22] milimetric: thank you for the heads up. [13:44:34] hashar: what do you mean? [13:45:17] I don't know I am lost [13:45:46] well. I think the longer we wait to stop gerrit, the more out of sync things are going to be. [13:45:48] Project beta-scap-eqiad build #272280: 04STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272280/ [13:46:03] yes [13:46:12] but I am struggle to find out how to detect out of sync changes :-\ [13:47:33] if stuff is all scrambled in the metadata db, I can help with some queries maybe? [13:48:49] milimetric: "db" is a loose term [13:48:52] it's all in git [13:49:37] hashar: we can say that all meta changes that are both in packed-refs and on disk are incorrect. So we should start with removing those. [13:49:55] I'm going to backup the current state of /srv/gerrit/git [13:49:58] well [13:50:28] those on disk created after the migration are probably correct [13:50:37] if they get created recently [13:50:45] oh [13:50:50] sorry I misparsed your sentence [13:51:08] so yeah if they are both in paked-refs and on disk they are out of sync [13:51:32] but potentially any changes made on the new server would be in the file [13:51:44] 10Gerrit: Patch statuses changed inappropriately from 'merged' or 'abandoned' back to 'open' - https://phabricator.wikimedia.org/T236165 (10Mholloway) [13:52:56] hashar: yes, that's true; however, there are likely fewer of those and we can deal with them on a case-by-case [13:54:04] maybe git has an utility to detect such conflictS? [13:54:52] Project beta-scap-eqiad build #272281: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272281/ [13:57:30] fun [13:57:34] if I pack-refs [13:57:38] if I git pack-refs [13:57:51] it dismissed the loose ref [13:58:10] and kept the one from the packed-refs file (which is up to date) [13:59:35] hmm [13:59:37] 10Gerrit: Patch statuses changed inappropriately from 'merged' or 'abandoned' back to 'open' - https://phabricator.wikimedia.org/T236165 (10Mholloway) [13:59:38] no the other way around [13:59:40] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Mholloway) [14:00:04] hashar: well git pack-refs is just going to move the files from disk into packed refs [14:00:10] does it keep the old value? [14:00:10] yeah [14:00:14] no [14:00:15] sorry [14:00:22] ah, bummer [14:00:41] so the meta file is outdated [14:00:58] and running ti packed-refs would move that meta file and consider it more up to date [14:01:03] and override the version in index-pack [14:01:28] so any repo that had the pack refs run for it (no idea when that happens in gerrit / jgit) would have an outdated version written in packed-refs [14:01:53] we run git gc on Friday only [14:03:33] hashar: for a given repo, this seems to be the fastest way to find these changes: find refs/changes -name meta | xargs -I{} grep "{}" packed-refs [14:04:00] thcipriani: hi [14:04:47] hashar: find refs/changes -name meta | xargs -I{} grep "{}" packed-refs | awk '{print $2}' | xargs rm [14:04:50] would probably work [14:04:55] paladox: hello [14:05:17] +1 to your work after doing a quick scroll in the back log [14:05:36] *suggestion [14:06:54] looks on copy of mediawiki-conifig [14:09:05] oh [14:09:06] but [14:09:10] packed-refs is not a binary file! [14:10:15] what about checking whether the meta file sha1 matches the one from packed-refs ? [14:10:23] if they match, we can safely delete the meta file for sure [14:10:49] if they mismatch, well we gotta dig into it [14:13:46] Yippee, build fixed! [14:13:46] Project beta-scap-eqiad build #272282: 09FIXED in 9 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272282/ [14:17:35] Project beta-scap-eqiad build #272283: 04FAILURE in 2 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272283/ [14:17:54] ah hmm [14:18:01] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10dbarratt) [14:19:56] thcipriani: find refs/changes -name meta| xargs -I{} bash -c 'grep "$(cat {}) {}" packed-refs' [14:20:06] that finds meta file that matches index-pack [14:20:10] and can be safely disposed [14:22:25] for the others we will need to compare the sha1 [14:23:35] https://gist.github.com/thcipriani/39f5a08139eb17998386a32f2145a8b3 [14:23:42] hashar: ^ [14:24:05] heh, missing a grep step there [14:24:33] but those meta files might be more up to date [14:24:40] right, hang on [14:24:42] than the one in the index pack [14:25:06] I experimented something which is [14:25:11] Project beta-scap-eqiad build #272284: 04STILL FAILING in 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272284/ [14:25:18] for each meta file, read the sha1 and craft a git tag for it [14:25:30] then for each packed ref in packed-refs, create another tag [14:25:37] then use git log --graph to inspect [14:26:01] or maybe git tag --contains to check whether the sha1 from the meta file is correctly contained by the ref in packed-refs [14:26:36] https://gist.github.com/thcipriani/39f5a08139eb17998386a32f2145a8b3 [14:28:41] yeah so those would be mismatching refs [14:29:06] but do you know which are the right refs? [14:29:11] and git cherry might help [14:29:15] well [14:29:22] that depends :-]]] [14:29:24] hashar: what do you think about running my script? [14:29:37] the ref in packed-refs might be the correct one if stuff got changed on cobalt firt [14:29:38] replace the "echo" with "rm" here: https://gist.github.com/thcipriani/39f5a08139eb17998386a32f2145a8b3 [14:29:43] and never changed on gerrit1001 after migration [14:29:56] or the correct one might be the file on gerrit1001 if it got changed after the migration [14:30:01] or there could be conflict [14:30:12] so we can't just rm them [14:30:12] ok, so if the mtime of the meta file > mtime of the migration [14:30:17] don't rm [14:30:19] otherwise rm [14:30:22] nop [14:30:23] thoughts on that? [14:30:36] cause you can get a weird condition [14:30:57] I think that weird conditions are unavoidable? [14:31:11] but there are probably just a handful of them [14:31:50] but git cherry can help figure out whether a sha1 is included in the others [14:32:09] but I have trouble to represent each of the case that might have happened [14:38:41] Project beta-scap-eqiad build #272285: 04STILL FAILING in 4 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272285/ [14:39:47] 10Deployments, 10Release-Engineering-Team, 10serviceops, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Jdforrester-WMF) How reliable is the `filemtime` function in a scap world? Does scap / rsync always migrate the mtim... [14:41:54] (03PS2) 10Jforrester: layout: [at-ease] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544819 [14:41:59] (03CR) 10Jforrester: [C: 03+2] layout: [at-ease] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544819 (owner: 10Jforrester) [14:42:09] (03PS2) 10Jforrester: layout: [base-convert] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544820 [14:42:11] (03CR) 10Jforrester: [C: 03+2] layout: [base-convert] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544820 (owner: 10Jforrester) [14:42:18] (03PS2) 10Jforrester: layout: [css-sanitzer] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544821 [14:42:22] (03CR) 10Jforrester: [C: 03+2] layout: [css-sanitzer] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544821 (owner: 10Jforrester) [14:42:28] (03PS2) 10Jforrester: layout: [AhoCorasick] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544822 [14:42:33] (03CR) 10Jforrester: [C: 03+2] layout: [AhoCorasick] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544822 (owner: 10Jforrester) [14:42:41] (03PS2) 10Jforrester: layout: [RunningStat] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544823 [14:42:46] (03CR) 10Jforrester: [C: 03+2] layout: [RunningStat] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544823 (owner: 10Jforrester) [14:42:52] (03PS2) 10Jforrester: layout: [IPSet] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544824 [14:42:56] (03CR) 10Jforrester: [C: 03+2] layout: [IPSet] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544824 (owner: 10Jforrester) [14:43:56] (03Merged) 10jenkins-bot: layout: [at-ease] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544819 (owner: 10Jforrester) [14:44:03] (03Merged) 10jenkins-bot: layout: [base-convert] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544820 (owner: 10Jforrester) [14:44:32] (03Merged) 10jenkins-bot: layout: [css-sanitzer] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544821 (owner: 10Jforrester) [14:44:39] (03Merged) 10jenkins-bot: layout: [AhoCorasick] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544822 (owner: 10Jforrester) [14:44:41] Project beta-scap-eqiad build #272286: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272286/ [14:44:42] (03Merged) 10jenkins-bot: layout: [RunningStat] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544823 (owner: 10Jforrester) [14:45:45] (03Merged) 10jenkins-bot: layout: [IPSet] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544824 (owner: 10Jforrester) [15:01:12] Yippee, build fixed! [15:01:12] Project beta-scap-eqiad build #272287: 09FIXED in 6 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/272287/ [15:24:44] (03PS2) 10Jforrester: layout: [RelPath] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544825 [15:24:50] (03CR) 10Jforrester: [C: 03+2] layout: [RelPath] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544825 (owner: 10Jforrester) [15:24:57] (03PS2) 10Jforrester: layout: [WrappedString] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544826 [15:25:01] (03CR) 10Jforrester: [C: 03+2] layout: [WrappedString] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544826 (owner: 10Jforrester) [15:25:11] (03PS2) 10Jforrester: layout: [CLDRPluralRuleParser] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544827 [15:25:16] (03CR) 10Jforrester: [C: 03+2] layout: [CLDRPluralRuleParser] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544827 (owner: 10Jforrester) [15:25:43] (03PS2) 10Jforrester: layout: [mediawiki/libs/Assert] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544830 [15:25:47] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/libs/Assert] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544830 (owner: 10Jforrester) [15:26:38] (03Merged) 10jenkins-bot: layout: [RelPath] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544825 (owner: 10Jforrester) [15:26:51] thcipriani: did you manage to run the command? [15:27:11] (03Merged) 10jenkins-bot: layout: [WrappedString] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544826 (owner: 10Jforrester) [15:27:17] (03Merged) 10jenkins-bot: layout: [CLDRPluralRuleParser] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544827 (owner: 10Jforrester) [15:27:41] (03Merged) 10jenkins-bot: layout: [mediawiki/libs/Assert] Drop PHP7.0/PHP7.1 testing [integration/config] - 10https://gerrit.wikimedia.org/r/544830 (owner: 10Jforrester) [15:45:54] (03PS11) 10Awight: Commands can expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 [15:45:56] (03PS1) 10Awight: Move class state into local variables [integration/quibble] - 10https://gerrit.wikimedia.org/r/545309 [15:46:38] (03CR) 10jerkins-bot: [V: 04-1] Commands can expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 (owner: 10Awight) [15:47:44] hashar: Good luck, sir1 [15:52:54] 10Gerrit, 10Operations: Editing in Gerrit isn't saved after the update/migration to gerrit1001 - https://phabricator.wikimedia.org/T236143 (10MoritzMuehlenhoff) This happened earlier the day, but I cannot currently reproduce it with a freshly created patch. [15:54:43] 10Gerrit: Increase GC runs for jgit - https://phabricator.wikimedia.org/T236182 (10Paladox) [16:15:08] James_F: About extensions claiming to support old MW/PHP: how are we going to behave re the PHPUnit migration? [16:18:16] PROBLEM - SSH access on gerrit1001 is CRITICAL: connect to address 208.80.154.137 and port 29418: Connection refused https://wikitech.wikimedia.org/wiki/Gerrit [16:18:20] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - page size 1529 too small - 1529 bytes in 0.011 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [16:18:26] PROBLEM - Check systemd state on contint2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:18:54] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1529 bytes in 0.008 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [16:19:44] PROBLEM - Check systemd state on contint1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:20:18] ACKNOWLEDGEMENT - SSH access on gerrit1001 is CRITICAL: connect to address 208.80.154.137 and port 29418: Connection refused daniel_zahn WIP https://wikitech.wikimedia.org/wiki/Gerrit [16:21:30] RECOVERY - SSH access on gerrit1001 is OK: SSH OK - GerritCodeReview_2.15.14-16-g855b179b5f (SSHD-CORE-1.6.0) (protocol 2.0) https://wikitech.wikimedia.org/wiki/Gerrit [16:21:32] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 27019 bytes in 0.075 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [16:21:40] RECOVERY - Check systemd state on contint2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:22:06] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 864 bytes in 0.044 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [16:22:58] RECOVERY - Check systemd state on contint1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:01:15] Daimona: We can't. They'll just be lost to time. [17:02:02] What I thought [17:02:15] At least for extensions like MWFarm that claim to support ancient versions [17:02:23] But what about reasonable cases like Translate? [17:02:34] Where they try to keep compat with supported MW versions [17:04:17] 10Gerrit: SSH login to gerrit no longer works for me (simetrical/Aryeh Gregor) - https://phabricator.wikimedia.org/T236191 (10Simetrical) [17:05:20] Daimona: I can't see a way to support their needs and ours. Maybe we're forced to wait? [17:06:30] There are ways, but they're not free [17:07:13] For instance, a phpunit4SetUp method in MWTestCase and similar [17:07:33] Then setUp would call that function, and extensions would override it, with no need to use setUp and its typehint [17:07:39] (03CR) 10Hashar: [C: 04-1] "I have been looking at Gerrit this afternoon for an issue post migration T236114 . So this change is delayed." [integration/config] - 10https://gerrit.wikimedia.org/r/544907 (https://phabricator.wikimedia.org/T188398) (owner: 10Hashar) [17:07:45] see you tomorrow! [17:07:54] The code will still require PHP71+ for testing, but at least it's not explicit in the extension's own code [17:11:47] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) [17:16:51] 10Continuous-Integration-Config, 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (201910), 10Scap, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) [17:20:30] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Mholloway) Should we be updating the description of this task with changes that need fixing, or will they all be fixed eventually regardless? [17:20:54] 10Gerrit: SSH login to gerrit no longer works for me (simetrical/Aryeh Gregor) - https://phabricator.wikimedia.org/T236191 (10Krenair) Did you try pushing over HTTPS? You can (or at least could, am on phone) get a password generated in Gerrit settings [17:33:57] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) tl;dr: we ran a script, it should have fixed a lot. It may not have fixed everything. **DO NOT** fiddle with the broken patches; e.g., rebase or +2 or merge or whatever. Please a... [17:34:24] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) >>! In T236114#5596069, @Mholloway wrote: > Should we be updating the description of this task with changes that need fixing, or will they all be fixed eventually regardless? If... [17:38:53] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Mholloway) [17:44:29] 10Gerrit, 10Release-Engineering-Team-TODO, 10Patch-For-Review: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Mholloway) [17:47:53] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:53] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:54:28] Project beta-code-update-eqiad build #268911: 04FAILURE in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/268911/ [17:55:25] 18:54:26 fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/skins/': The requested URL returned error: 503 [17:56:34] that'll be because gerrit's getting restarted [17:56:55] to change a config [17:56:56] (for https://phabricator.wikimedia.org/T236114) [17:57:47] 10Gerrit, 10Release-Engineering-Team-TODO, 10Patch-For-Review: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10AnneT) [17:58:34] Reedy: back [17:59:03] 10Gerrit: SSH login to gerrit no longer works for me (simetrical/Aryeh Gregor) - https://phabricator.wikimedia.org/T236191 (10Tgr) Gerrit bails out after five failed "logins" - that is, if you have more than five miscellaneous identity files preceding the right one. Yes, it's stupid. (We have a task about it but... [18:00:01] 10Gerrit: SSH login to gerrit no longer works for me (simetrical/Aryeh Gregor) - https://phabricator.wikimedia.org/T236191 (10Tgr) (Also FWIW you can push to gerrit via non-anonymous HTTP, with an app password you can generate in your settings.) [18:01:04] 10Gerrit, 10Release-Engineering-Team-TODO, 10Patch-For-Review: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10AnneT) [18:04:27] Yippee, build fixed! [18:04:28] Project beta-code-update-eqiad build #268912: 09FIXED in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/268912/ [18:08:15] 10Scap, 10observability, 10serviceops: logstash_checker should be able to check for error for any php version - https://phabricator.wikimedia.org/T235490 (10Krinkle) [18:08:30] 10Scap, 10observability, 10serviceops: logstash_checker should be able to check for error for any php version - https://phabricator.wikimedia.org/T235490 (10Krinkle) Decoupling from the PHP 7.2 migration deliverable. [18:09:06] mutante thcipriani https://wikitech.wikimedia.org/w/index.php?title=Gerrit&type=revision&diff=1841934&oldid=1840631 :) [18:11:40] paladox: thank you! especially the part to mention --delete. that's what i had in mind so we see that in (2 years?) when we migrate from gerrit1001 to 1003 [18:11:50] yup [18:11:59] i say 1003 because i want 1002 to be standby later [18:12:07] also on buster [18:12:44] :) [18:12:46] 10Gerrit: SSH login to gerrit no longer works for me (simetrical/Aryeh Gregor) - https://phabricator.wikimedia.org/T236191 (10Simetrical) >>! In T236191#5596199, @Tgr wrote: > Gerrit bails out after five failed "logins" - that is, if you have more than five miscellaneous identity files preceding the right one. Y... [18:13:24] paladox: this is the actual "DNS hack" https://tools.wmflabs.org/sal/log/AW3wKZvNfYQT6VcDX2pS [18:13:38] * paladox links [18:14:05] done [18:14:09] hopefully those SAL links stay forever [18:14:12] thx [18:14:49] yup [18:16:47] paladox: actual example of rsync commandline.. without --delete but to show the syntax since it's not over ssh [18:16:51] rsync -avp /srv/gerrit/git/operations/puppet.git/objects/ rsync://gerrit1001.wikimedia.org/gerrit-data/git/operations/puppet.git/objects/ [18:17:04] wait.. that was just git objects for one repo [18:17:29] thanks! [18:17:36] Should i document that? [18:17:43] (the rsync command) [18:17:49] paladox: so in this example "gerrit-data" is the name of the "module" [18:17:56] not a file system path [18:17:58] that's the gotcha [18:18:00] yea [18:18:02] ah [18:18:03] or i can [18:18:15] mutante thanks, yes please!! [18:19:13] gerrit-reivew has zuul running! https://gerrit-review.googlesource.com/c/plugins/checks/+/241293/1#message-e42933683fca0d537192d99d3dee741bb846bff9 [18:19:49] (opendev's running that) [18:26:33] https://wikitech.wikimedia.org/w/index.php?title=Gerrit&type=revision&diff=1841938&oldid=1841937 [18:27:15] thcipriani: ^ rsync commands for next time .. right [18:28:31] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) [18:28:53] did we finish " Run the online reindexer" ? [18:32:05] mutante: yeah, I've run it again today [18:32:49] ack, cool! [18:33:00] paladox: removing "Also rsync lfs objects again." because that is part of the existing rsync commands [18:33:11] ok! thanks! [18:33:19] would cause confusion about where those are [18:34:37] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) [18:36:07] https://wikitech.wikimedia.org/w/index.php?title=Gerrit&type=revision&diff=1841940&oldid=1841938 [18:36:24] thanks mutante ! [18:38:18] paladox: i guess it comes down to "create a patch like..." for all of this? https://gerrit.wikimedia.org/r/q/topic:%22gerrit1001%22+(status:open%20OR%20status:merged) [18:38:28] like when we added new IPs and everything else [18:38:34] yes! [18:38:56] except that we dont have to recreate the gerrit::migration class and some other things that were mistakes [18:39:19] linking the topic branch..that's what it was for, so have a single URL [18:41:43] thcipriani https://bugs.chromium.org/p/gerrit/issues/detail?id=11650#c2 \o/ [18:42:21] i think we should revert the "integrate migration class into main class" thing? [18:42:55] if we keep it and gerrit1001 is also the master_host [18:43:03] then there is rsync setup to allow pushing to ..itself [18:43:31] the next time we migrate and have for example gerrit1003.. then we would start again by putting only the migration class on it [18:43:49] and that would allow pushing from gerrit1001 TO gerrit1003 [18:44:05] except then after that first phase.. we want to go back to "both roles in one" [18:45:14] or we keep it like this all the time.. where rsyncd runs but only allows connections from the host itself [18:45:39] Could we integrate it but put it behind a flag? [18:46:10] oh heh gerrit mail is very slow with gerrit... [18:46:31] *Yahoo [18:46:34] just got a notification about me deleting and adding a ssh key [18:46:35] paladox: flag to include or not include the whole class? yes, we could [18:46:44] yup [18:47:31] hashar https://bugs.chromium.org/p/gerrit/issues/detail?id=11650#c2 :) [18:50:30] paladox: great :] [18:50:49] paladox: are we _really_ removing the whole avatars thing in general? [18:50:56] yeh for now [18:51:32] declined per https://phabricator.wikimedia.org/T191183#4861609 ? [18:51:45] i did not actually mean to start that discussion [18:51:53] i just dont want the dependency for apache [18:51:59] that was the important part [18:52:12] that apache isnt broken just because it cant clone avatars from itself [18:52:58] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Marostegui) [18:53:24] mutante not really, needs the software to be inplace to upload/optimise. [18:56:15] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10International-Developer-Events, 10Wikimedia-Technical-Conference-2019, and 2 others: Wikimedia Technical Conference 2019 Session: System level testing: patterns an... - https://phabricator.wikimedia.org/T234635 [18:56:19] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) We have caught any change for which we still had a `meta` file on gerrit1001. But for some repositories on `gerrit1001`, the `meta` files had already been packed in `packed-refs` a... [18:56:30] thcipriani: so looks like some meta file had already been packed :-\\ [18:59:46] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) [19:00:25] puppet is still disabled on gerrit2001 [19:01:20] removing the avatars stuff, ack paladox [19:04:01] mutante thanks [19:06:32] 10Gerrit, 10Operations, 10Traffic, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10Dzahn) ^ The reason to merge this was not a comment on the general question to enable avatars. The reason was that during T222391 we noticed an undesirable dependency. During a Ger... [19:06:51] paladox: i left a lengthy comment there to explain [19:06:57] ok [19:07:03] thanks! [19:07:52] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) [19:09:45] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10Dzahn) This is mostly done and all boxes are checked. Though only re... [19:14:50] 10Continuous-Integration-Infrastructure, 10Toolforge, 10Kubernetes, 10cloud-services-team (Kanban): Add CI checks for golang admission controllers - https://phabricator.wikimedia.org/T236203 (10Bstorm) [19:15:07] 10Continuous-Integration-Infrastructure, 10Toolforge, 10Kubernetes, 10cloud-services-team (Kanban): Add CI checks for golang admission controllers - https://phabricator.wikimedia.org/T236203 (10Bstorm) a:05Bstorm→03None [19:15:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Toolforge, and 2 others: Add CI checks for golang admission controllers - https://phabricator.wikimedia.org/T236203 (10hashar) [19:16:52] https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=activeThreads&period=mois <3 [19:17:00] threads looking healthier! [19:17:19] and it's only been a day :P [19:18:07] :) [19:21:50] * paladox waiting for uk gov to table an election! [19:23:18] a quote about heap_size in Gerrit: [19:23:22] " According to Oracle documentation, starting from J2SE 5.0 it is the smaller of 1/4th of the physical memory or 1GB " [19:23:31] this does not seem to make sense [19:23:48] it does not match the next sentence which is "We recommend you set this parameter to 8GB for a 16GB machine and 16GB or more for a machine with 24GB or more memory. " [19:24:00] I think we should try 32GB (which is half of 64) [19:24:03] that first sentence would mean always set it to 1GB, because that's the smaller value [19:24:24] oh [19:24:25] paladox: for once i already have the patch before you :) [19:24:25] https://gerrit.wikimedia.org/r/c/operations/puppet/+/545381 [19:24:32] mutante was about to say :D [19:24:33] mutante: hi :) the rsync server still on cobalt, that requires ssh isn't it ? [19:24:33] lol [19:24:34] 32GB..was my thinking too [19:24:43] it is not an anonymous rsync server is it [19:25:26] oh [19:25:37] RECOVERY - Puppet staleness on deployment-cache-text05 is OK: OK: Less than 1.00% above the threshold [3600.0] [19:25:41] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:25:41] hosts allow dbmonitor1001.wikimedia.org. [19:25:43] so hmm [19:25:45] not allowed :] [19:25:53] hashar: it's not rsync over ssh, it's just rsync. there are 2 places where it limits allowed hosts. rsyncd and ferm. [19:26:06] and we pushed from cobalt to the new server [19:26:10] not the other way around [19:26:29] and it is firewalled off [19:26:32] yeah [19:26:34] ;D [19:26:56] the question is how long we wait before we remove the role from cobalt [19:27:03] and also we want a new patch that adds a parameter [19:27:19] we need to keep it as is for now [19:27:25] to turn the rsyncd setup on and off. since now we unified the roles for "regular gerrit" and "gerrit migration" [19:28:35] hashar: so did you ask because you want to prevent rsync or you want to rsync again .. or ? [19:28:41] no no [19:28:55] I just though that maybe I could use rsync to pull data from cobalt to gerrit1001 [19:29:01] I wanted to sync a repo. Then it is just 1.5MB [19:29:12] so I have just created a tarball and transfered it via my home connection ;D [19:29:50] hashar: it would work but you got to upload from cobalt to gerrit1001 [19:29:53] problem solved :] [19:30:02] but also.. does it mean something was missing? [19:30:05] in the docs [19:30:08] I have just scp cobalt [19:30:13] then scp gerrit1001 [19:30:19] for 1.5MB that is good enough [19:30:25] na [19:30:41] it is because some references are missing ( https://phabricator.wikimedia.org/T236114 ) [19:30:48] so I am trying to check one of the affected repo [19:31:11] ok, ACK. let me know if you need more syncing to something [19:31:19] sure :) [19:31:35] and I shared you a doc from this afternoon debug session. Nothing really interesting, it is more of a brain dump [19:32:48] thanks. i just got notified about that. should we merge the info into a single place? [19:33:52] hey mutante im seeing Gerrit patches that have been merged coming back from the dead. I assume this is known but wondered if it's safe to abandon? https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/537742/ [19:34:09] ^ this patch is in master but gerrit interface is suggesting it never merged - caused us some confusion this morning [19:34:43] hashar: we made https://wikitech.wikimedia.org/wiki/Gerrit#Migrating for next time. thanks for all the work on the fixes [19:35:24] Jdlrobson: don't abandon it, just add the link to https://phabricator.wikimedia.org/T236114 please [19:35:45] it should eventually come back as normal [19:36:13] hashar: ^ is that right? [19:36:40] yeah [19:36:53] Jdlrobson: we are tracking them down :\ [19:37:34] Jdlrobson: there are probably more changes affected in MobileFrontend [19:37:42] I think we only got some of the repositories fixed [19:37:53] but missed a bunch of repositories due to lack of data/infos on gerrit1001 [19:38:42] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Jdlrobson) [19:38:54] k! done! thanks for the info! [19:38:58] might be worth a wikitech email [19:41:25] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Jdlrobson) [19:41:39] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:44:14] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10nray) [19:45:35] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10nray) [19:46:40] 10Gerrit, 10Release-Engineering-Team-TODO, 10Patch-For-Review: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) [19:46:50] hashar thcipriani https://gerrit-review.googlesource.com/c/plugins/delete-project/+/242132 [19:46:52] \o/ [19:51:09] hashar: thcipriani: there is no reason you wanted puppet disabled on gerrit2001, right? it's just me who did not reactivate it again? [19:52:01] no idea [19:54:16] Jdlrobson: you are right, mailed wikitech [19:58:30] mutante: I did not want it disabled, no [19:59:44] thcipriani: ACK, re-enabling [20:00:28] that also disabled gc there [20:03:47] 10Gerrit, 10Release-Engineering-Team-TODO, 10Patch-For-Review: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) I looked at the MachineVision repo change 538339. I have made a copy of the repo from cobalt to gerrit1001 and then compared the references. And indeed we are... [20:04:34] I would guess we can use a copy of cobalt/srv/gerrit/git to be made available on gerrit1001 [20:04:44] so we can "easily" transfer the missing metadata [20:05:50] 10Gerrit, 10Release-Engineering-Team-TODO, 10Patch-For-Review: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) [20:05:50] hashar: ok, hold on [20:06:25] there are least two repositories affected MachineVision and MobileFrontend [20:06:34] and there must be some others [20:06:52] I guess some how the loose refs have been garbage collected somehow [20:06:58] ^ [20:07:11] yeah, too many loose objects causing ref packing [20:07:13] is my guess [20:07:18] yup [20:07:29] cause they don't show in the script output from earlier today [20:07:42] exactly [20:08:03] I did some rough comparison for MachineVision and it is really the same case [20:08:19] and potentially there could be split brain cases (haven't checked) [20:08:51] yeah, I think the same thing probably happened, i.e., shadow refs/* files, but pack-refs being triggered makes this more complicated to solve [20:09:01] 10Gerrit, 10Operations: Editing in Gerrit isn't saved after the update/migration to gerrit1001 - https://phabricator.wikimedia.org/T236143 (10colewhite) p:05Triage→03Normal [20:09:02] then now we now we "just" need to use git merge-base :] [20:09:21] so I have a list of repos now that have been modified since Oct 1st [20:09:40] and I have compared that with repos that we "fixed"; i.e., that were in our script output [20:09:41] #!log gerrit1001 - mkdir /srv/gerrit/cobalt/git - rsyncing /srv/gerrit/git from cobalt to /srv/gerrit/cobalt/git/ on gerrit1001 (T236114) [20:09:41] T236114: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 [20:10:26] thcipriani well done! [20:10:48] rsync is ongoing [20:10:50] mutante: awesome. This way we have everything we need on the same host ;] [20:11:13] * mutante checks disk space and /srv has only 35% usage [20:11:13] so this is the list of potentially affected repos: https://phabricator.wikimedia.org/P9440 [20:12:22] next step, narrowing that list further [20:12:42] some we got them fixed as part of the script earlier today [20:12:49] uhm.. including All-Projects/All-Users ? [20:12:52] yep, those have been removed from this list [20:13:14] what I thought is: [20:13:24] mutante: those are just repos that have been updated since Oct1 that weren't in our script output. Doesn't mean too much at this point. [20:13:25] a) clone all repositories from /srv/gerrit/git [20:13:45] b) for each repo, git remote add cobalt /srv/gerrit/cobalt/git/$REPO [20:13:59] and then for each repo run a script that compare the refs [20:14:19] which refs? [20:14:41] eg whenever a meta file is different, check wether the one from cobalt is a descendant or if we have split brain [20:14:48] refs/changes/*/*/meta I guess [20:14:54] same as what we did earlier today [20:15:17] but this time using the references from cobalt insead of comparing packed-refs versus meta files [20:15:48] hrm, well, earlier we had a narrower search space: looped through all /meta refs on disk as files in the ref/ dir [20:16:24] (if we had all of that in a graph database and a couple graph theory phd student that would be solved with a single query! ) [20:16:28] now it's...meta refs updated since oct-1st? [20:17:40] yeah [20:17:44] mhmm [20:17:44] git log --all --format=%H%d --after=2019-10-01 | grep -P 'meta$' [20:18:05] so earlier we wanted to resolve the disprendency between the stall info due to the meta files [20:18:06] rsync is done [20:18:11] and compared packed-refs with meta files [20:18:12] see /srv/gerrit/cobalt now [20:18:28] but since some of those meta disappeaered / got packed [20:18:48] we now need to resolve the dispredancy between cobalt original repo and gerrit1001 [20:18:53] mutante: that was fast thx :) [20:18:56] 59,494,487.24 bytes/sec [20:19:14] 30 GB [20:20:08] and in theory the gerrit fix3 script from earlier should not report much issues if run against /srv/gerrit/cobalt [20:24:10] ok, so, loop through all repos, loop through all meta refs after 2019-10-01, find merge base between cobalt and gerrit1001 for each meta ref...then... [20:24:20] fun my ISP has lost both dns recursors [20:24:36] then compare the sha1 using git-merge base [20:25:06] if the one from cobalt is a children from the one on gerrit1001, it is more up to date [20:25:26] if the merge-base is cobalt, update-ref on gerrit1001... [20:25:46] and if merge-base == gerrit1001: normal [20:25:53] sound right? [20:26:04] and if the merge base is neeither gerrit1001 nor cobalt: SPLIT BRAIN ! [20:26:08] right [20:26:16] and the direction gotta be double checked [20:26:27] since it took us a few minutes to ensure it was right [20:26:57] I think one only has to compare refs/changes/*/*/meta references [20:27:09] if they point to the same sha1 ... can be skipped entirely [20:27:31] I looked at MachineVision by just diffing the output of git ls-remote . [20:35:29] thcipriani: MachineVision.git is super small (1.5MB) and has ~ 10 broken changes [20:35:35] so maye it is a good sandbox area [20:53:33] 10Gerrit, 10Release-Engineering-Team-TODO: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Tchanders) [21:17:25] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) p:05High→03Unbreak! a:03thcipriani [21:51:09] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10hashar) @thcipriani and @hashar paired again on a script to catch the last affected repositories/changes... [22:35:08] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) [22:43:20] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10thcipriani) 05Open→03Resolved >>! In T236114#5596944, @hashar wrote: > @thcipriani and @hashar paire... [22:43:24] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Gerrit Hardware Upgrade (+ upgrade from jessie to stretch or buster) - https://phabricator.wikimedia.org/T222391 (10thcipriani) [23:06:12] (03PS1) 10Paladox: Update scap targets [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/545416 [23:06:23] (03PS2) 10Paladox: Update scap targets [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/545416 [23:11:10] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Mholloway) For the affected MachineVision changes, it looks like the full histories are restored, but th... [23:17:56] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Wikimedia-Incident: check and fix some Gerrit revs - https://phabricator.wikimedia.org/T236114 (10Paladox) @Mholloway should be resolved now, @thcipriani started the online reindexer.