[03:14:17] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Winston_Sung) >>! In T337427#8878150, @gerritbot wrote: > Change 922925 had a...
[07:27:22] <wikibugs>	 (03CR) 10Hashar: Enable API tests for Flow (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[07:35:29] <wikibugs>	 (03PS2) 10Hashar: Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[07:36:39] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[07:36:43] <wikibugs>	 (03CR) 10Hashar: Enable API tests for Flow (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[07:40:40] <wikibugs>	 (03PS3) 10Hashar: Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[07:53:29] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @nshahquinn-wmf: Hi, basically [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) - if privileges have not been used for a long whil...
[07:57:53] <hashar>	 TheresNoTime: I eventually reached https://people.wikimedia.org/~samtar/  and that is probably the best homepage I have ever seen on people.wm.o :D
[07:58:18] <TheresNoTime>	 hashar: aw thank you :D
[07:58:38] <hashar>	 https://people.wikimedia.org/~hashar/ :D
[07:58:41] <hashar>	 it is pale in comparison
[08:00:05] <TheresNoTime>	 `~sites` are meant to be frugal though! It serves a purpose ^^
[08:21:04] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10hashar) p:05Triage→03Unbreak! Train blockers are {nav Unbreak Now!} priority.
[08:23:49] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Winston_Sung) > https://meta.wikimedia.org/w/index.php?banner=wle_2023_tw&camp...
[08:25:28] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) >>! In T337427#8879085, @hashar wrote: > Train blockers are {nav Un...
[08:30:29] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) I don't think this should be a blocker, it stopped right after the...
[08:32:43] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func) But we should have a stable way to invalidate the cache after new conver...
[08:38:51] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) >>! In T337427#8879138, @Func wrote: > But we should have a stable...
[09:47:36] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[09:57:36] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[10:21:32] <duesen>	 hashar: do you have time for a quick call in half an hour or so to sort out the dependencies around the API tests for FLow?
[10:21:39] <duesen>	 I think they are the wrong wy now...
[11:13:59] <wikibugs>	 (03PS4) 10Daniel Kinzler: Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833
[11:14:08] <wikibugs>	 (03CR) 10Daniel Kinzler: Enable API tests for Flow (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[11:16:39] <duesen>	 hashar: I flipped the dependencies again... well, I guess either order would work. In reality, both patches should go in together. If we merge the zuul patch first, CI will be broken until the tests are added to the repo. If we mrege the tests first, we don't know if they will actually pass in CI, so again, CI for the repo may be broken.
[11:19:53] <duesen>	 But jsduck is choking... for no reason, as far as I can see.
[11:20:04] <duesen>	 Why is jsduck even enabled for the test directory?
[11:56:18] <hashar>	 duesen:  sorry I was having lunch + nap
[11:56:50] <hashar>	 so for the dependency usually we merge the CI config first
[11:57:15] <hashar>	 then CR+2 the other work which as a result is tested with the new job
[11:57:31] <hashar>	 hence why I have flipped the dependency, I should probably not have amended the patches though
[12:00:02] <hashar>	 jsduck is obsolete, supposedly frontend folks should have migrated to JSDoc by now but I don't think it will ever happen until we force it (by causing preventing merges on repo still using it)
[12:00:20] <hashar>	 but  I am ranting :]
[12:01:38] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10jnuche) 05Open→03Resolved
[12:02:14] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10jnuche)
[12:12:35] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10jnuche) In a project's `.gitlab-ci.yml`, it is now possible to publish documentation and test c...
[12:31:56] <duesen>	 hashar: ok, I'll adjust the jsduck config so it will ignore the new file. 
[12:32:26] <duesen>	 I don't care about the merge order... Let's just get it done :)
[12:35:51] <hashar>	 jsduck must be stuck in the past and not recognizing the new syntax :D
[12:39:31] <duesen>	 I was thinking that as well, but running it locally it works... and it seems to be the same version of jsduck...
[12:39:32] <duesen>	 whatever
[12:39:48] <duesen>	 i just removed the test directory from the jsduck config
[12:40:02] <duesen>	 we don't need docs for the tests, do we? 
[12:40:37] <wikibugs>	 10GitLab (CI & Job Runners): Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10jnuche)
[12:40:53] <duesen>	 hashar: the patch should pass CI in about 15 minutes. When it does, can you merge the zuul patch and then recheck?
[12:41:06] <hashar>	 duesen: for sure!
[12:41:09] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟): Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10jnuche) a:03jnuche
[12:41:15] <wikibugs>	 (03PS5) 10Daniel Kinzler: Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833
[12:41:16] <duesen>	 cool
[12:41:18] <hashar>	 this way you know your code is good :]
[12:42:06] <duesen>	 yes. and if it passes, can you +2 that patch as well? Derick is out sick, and Richika doesn't have +2
[12:43:52] <wikibugs>	 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10Ottomata)
[12:46:56] <hashar>	 duesen: YES
[12:47:26] <hashar>	 and surely if Richika is working for your team, they should have +2 on  mediawik?
[13:05:54] <duesen>	 hashar: she's an intern, and treated as a contractor. But I agree, she should.
[13:06:09] <duesen>	 hashar: the patches are green now, can you +2?
[13:11:35] <hashar>	 sure
[13:11:47] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[13:11:56] <hashar>	 duesen: let me deploy that one :)
[13:13:07] <wikibugs>	 (03Merged) 10jenkins-bot: Enable API tests for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/922833 (owner: 10Daniel Kinzler)
[13:15:13] <hashar>	 I did a recheck https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/922822
[13:15:28] <hashar>	 https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-php74-docker/25617/console
[13:15:40] <hashar>	 if that job passes, I will CR+2
[13:18:49] <hashar>	 of course it failed
[13:18:50] <hashar>	 grr
[13:19:04] <hashar>	 00:01:16.907   1) Flow conversion utilities API
[13:19:04] <hashar>	 00:01:16.907        will convert from wikitext to HTML:
[13:19:05] <hashar>	 00:01:16.907      AssertionError: expected '<div class="mw-parser-output"><h2><sp…' to match /<h2.*?>Foobar<\/h2>/s
[13:22:26] <hashar>	 mocha probably need to be passed some arg to avoid elipsis in the assert output
[13:24:20] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T330216 (10Func)
[13:24:22] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func)
[13:24:34] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func) p:05Unbreak!→03Triage Lower the priority and remove the blocker pare...
[13:42:53] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep Puppet CA about to expire on 2023-05-25 - https://phabricator.wikimedia.org/T335689 (10Jgiannelos) From one test instance on deployment-prep: ` Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 stat...
[13:43:20] <hashar>	 duesen: it is broken :/
[13:45:25] <duesen>	 interesting... it's broken in a "good" way, though
[13:45:28] <duesen>	 I'll fix the regex
[13:48:10] <wikibugs>	 (03PS1) 10Hslater: Move 'StandardDialog' from 'phan_dependencies' to 'dependencies' [integration/config] - 10https://gerrit.wikimedia.org/r/923343
[13:51:57] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Allow pasting an image in a task - https://phabricator.wikimedia.org/T77020 (10Esanders) 05Open→03Resolved a:03Esanders This is possible.
[14:26:25] <duesen>	 hashar: it's geen now: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/922822/11
[14:30:17] <hashar>	 will +2 it in roughly 20 minutes
[14:33:51] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟): Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10jnuche)
[14:56:03] <hashar>	 duesen: +2ed 
[14:57:05] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func)
[14:57:08] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T330216 (10Func)
[14:57:12] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func) Wait - no more errors because they reverted group 2 to the wmf.9 branch.
[14:57:59] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func) p:05Triage→03Unbreak!
[15:01:25] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) Group2 never got wmf.10. It will get it later today. Group1 was dep...
[15:05:30] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10Patch-For-Review, 10Release, and 2 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func) Then why the trace is from wmf.10, I didn't get it.  Anyway, I can repro...
[15:14:39] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1001 for host gerrit2002.wikimedia.org with OS bullseye
[15:17:23] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Clement_Goubert)
[15:17:47] <wikibugs>	 10Phabricator, 10Developer-Advocacy (Apr-Jun 2023), 10Patch-For-Review: Automate SQL queries for annual end-of-year Phabricator statistics/metrics on wikitech-l@ - https://phabricator.wikimedia.org/T337388 (10Aklapper) a:03Aklapper
[15:17:53] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Clement_Goubert) p:05Triage→03High
[15:18:03] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Clement_Goubert)
[15:19:41] <hashar>	 mutante: Puppet will install Gerrit via scap::target and should boot it up
[15:19:45] <hashar>	 once puppet as ran I can check it
[15:19:46] <hashar>	 ;)
[15:21:01] <mutante>	 'spicerack.remote.RemoteHosts.wait_reboot_since' raised: Reboot for gerrit2002.wikimedia.org not found yet, keep polling for it: unable to get uptime
[15:21:04] <mutante>	 ...
[15:21:10] <mutante>	 waits
[15:21:27] <mutante>	 Caused by: Cumin execution failed (exit_code=2)
[15:22:17] <mutante>	 ok. it's working regardless 
[15:22:18] <mutante>	 Host up (Debian installer)
[15:26:12] <wikibugs>	 10Continuous-Integration-Config, 10Toolhub, 10Documentation, 10Patch-For-Review, 10User-Slst2020: Set up publish-docs pipeline job - https://phabricator.wikimedia.org/T308632 (10bd808) >>! In T332953#8879758, @jnuche wrote: > In a project's `.gitlab-ci.yml`, it is now possible to publish documentation an...
[15:27:03] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralAuth, 10Patch-For-Review, and 3 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func)
[15:27:08] <hashar>	 mutante: yes the raise is normal
[15:28:50] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review, and 3 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Winston_Sung)
[15:29:08] <wikibugs>	 10Release-Engineering-Team, 10Scap: "Not running from a virtual environment." warning in generated scap docs - https://phabricator.wikimedia.org/T337493 (10dancy)
[15:30:28] <wikibugs>	 (03PS1) 10Ahmon Dancy: Update scap URLs [integration/docroot] - 10https://gerrit.wikimedia.org/r/922540 (https://phabricator.wikimedia.org/T336168)
[15:32:33] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+1] Update scap URLs [integration/docroot] - 10https://gerrit.wikimedia.org/r/922540 (https://phabricator.wikimedia.org/T336168) (owner: 10Ahmon Dancy)
[15:39:44] <mutante>	 it is back from installer and starting first puppet run
[15:39:56] <mutante>	 this is likely longer than the OS install
[15:41:05] <mutante>	 also if puppet code isnt perfect to make everything work on first run.. it will fail. many roles require 2 runs though and doesnt matter except for the first few minutes of a server's new life
[15:41:57] <thcipriani>	 I'll be curious to see the scap installer for gerrit works on the first try
[15:42:08] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Update scap URLs [integration/docroot] - 10https://gerrit.wikimedia.org/r/922540 (https://phabricator.wikimedia.org/T336168) (owner: 10Ahmon Dancy)
[15:42:47] <wikibugs>	 (03Merged) 10jenkins-bot: Update scap URLs [integration/docroot] - 10https://gerrit.wikimedia.org/r/922540 (https://phabricator.wikimedia.org/T336168) (owner: 10Ahmon Dancy)
[15:46:34] <mutante>	 configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
[15:46:43] <mutante>	 Rebooting 1 hosts in batches of 1 with 0.0s of sleep in between: gerrit2002.wikimedia.org
[15:47:34] <mutante>	 I got shell
[15:47:48] <mutante>	 PuppetHosts.wait_since' raised: Unable to find a successful Puppet run
[15:48:00] <mutante>	 [gerrit2002:~] $ lsb_release -c
[15:48:01] <mutante>	 Codename:	bullseye
[15:48:38] <mutante>	 now I can try run puppet myself.. while the cookbook says it couldnt find a succesful run yet.. but is not giving up
[15:49:09] <mutante>	 [gerrit2002:/srv/deployment/gerrit/gerrit-cache] $ file cache/
[15:49:10] <mutante>	 cache/: directory
[15:49:24] <mutante>	 thcipriani: ^ looks like it deployed
[15:49:37] <thcipriani>	 ^ hashar 
[15:49:44] <mutante>	 runs puppet manually 
[15:49:57] <hashar>	 I will check ;)
[15:50:00] <mutante>	 leaves the cookbook sitting 
[15:50:04] <thcipriani>	 <3
[15:50:10] <hashar>	 but I am pretty sure Puppet should bring it up for us
[15:50:13] <mutante>	 Error: Execution of '/usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False' returned 70: 
[15:50:16] <mutante>	 ahh
[15:50:21] <mutante>	 so that is why it wasnt succesful
[15:50:33] <mutante>	 I bet it's that "tin" or something is in there again
[15:50:40] <mutante>	 remember that issue
[15:50:41] <hashar>	 with a sequence like scap::target["gerrit/gerrit"]  >  bunch of config >  Service["gerrit"] { ensure => running }
[15:50:55] <mutante>	 it's every time we add on a fresh host and it's deployed via scap
[15:50:57] <hashar>	 what is the log?
[15:51:46] <mutante>	 there is one file in the /srv/deployment/ dir that always has the deployment server
[15:51:50] <mutante>	 remember which one it was?
[15:52:09] <mutante>	 the problem was always that it had the name of the previous deployment server in it
[15:52:16] <mutante>	 looking for that
[15:52:42] <mutante>	 https://phabricator.wikimedia.org/T257317
[15:52:54] <mutante>	 looks at /srv/deployment/gerrit/gerrit-cache/.config 
[15:53:06] <mutante>	 has deploy1002.eqiad.wmnet in it
[15:53:15] <mutante>	 as git_server
[15:53:59] <mutante>	 here is this thing when we did gerrit1003 https://phabricator.wikimedia.org/T257317#8806038 though
[15:54:02] <thcipriani>	 I doubt that's the case here. That issue comes up when we haven't deployed something for a while.
[15:54:33] <mutante>	 running: sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False 
[15:54:45] <mutante>	 it works when I run it manually, like last time
[15:54:50] <mutante>	 re-running puppet agent
[15:54:51] <hashar>	 ...
[15:54:57] <hashar>	 did it exit 70?
[15:55:26] <mutante>	 sorry, I did't check the exit code before running the next command 
[15:55:35] <mutante>	 it showed no errors in text
[15:55:52] <mutante>	 everything is fixed in puppet now too
[15:56:07] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[15:56:08] <mutante>	 now the cookbook hopefully detects that succesful puppet run
[15:56:12] <mutante>	 and continues with other stuff
[15:56:16] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo) p:05Triage→03High
[15:57:32] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[15:57:34] <mutante>	 cookbook is increasing waiting period between each attempt.. was 30s, now 180s until next time
[15:57:52] <thcipriani>	 aside: the "tin" issue is that /srv/deployment/<repo>/<repo>-cache/.config comes from the current deployment host. And rather than just trust the current deployment host, it uses deploy-host:/srv/deployment/<repo>/<repo>/.git/DEPLOY_HEAD and that's generated by a deployment. So, if we switch deploy hosts after the last deployment, it's got outdated information.
[15:57:54] <mutante>	 Info: /Stage[main]/Gerrit/Systemd::Service[gerrit]/Service[gerrit]: Unscheduling refresh on Service[gerrit]
[15:58:01] <mutante>	 ^ this is good, service was started
[15:58:18] <mutante>	 Gerrit::Replication_key/Ssh::Userkey[gerrit-replication-publickey]/File[/etc/ssh/userkeys/gerrit2]/ensure: defined content
[15:58:18] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[15:58:56] <mutante>	 thcipriani: ACK, this makes sense why it was the case with gerrit1003 but not this time.. I did not have to edit the file. the difference is there was no DC switch in between
[15:59:16] <mutante>	 the "return with 70" seems separate 
[15:59:18] <thcipriani>	 yeah, I blame 2016 me, personally
[15:59:40] <mutante>	 cookbook DID notice puppet was succesful and is unstuck
[15:59:57] <mutante>	 it's checking icinga now for service recovery..it says
[16:00:37] <mutante>	 looking at icinga myself it's all green except the gerrit process
[16:00:42] <mutante>	 let me start that lfs rsync
[16:01:34] <mutante>	 rsync of lfs from gerrit1003 to gerrit2002 running with 100m limit
[16:01:54] <hashar>	 Gerrit will not start until the repos are synced in
[16:02:11] <mutante>	 hashar: service is in active/running
[16:02:27] <mutante>	 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gerrit2002.wikimedia.org with OS bullseye
[16:02:28] <hashar>	 fun
[16:02:31] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1001 for host gerrit2002.wikimedia.org with OS bullseye complet...
[16:02:37] <mutante>	 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gerrit2002.wikimedia.org with OS bullseye - Updated Phabricator task T334521
[16:02:37] <stashbot>	 T334521: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521
[16:03:49] <mutante>	 rescheduling the process check on icinga. it should just be outdated?
[16:03:59] <mutante>	 it's a very specific regex on the command line
[16:04:07] <mutante>	 but java-11 is still java-11
[16:04:37] <mutante>	 "0 processes with regex args '^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site' "
[16:04:59] <hashar>	 I have masked gerrit
[16:05:11] <mutante>	 ok
[16:05:14] <hashar>	 it is not going to start properly until All-Projects.git is replicated
[16:05:19] <mutante>	 the rsync of lfs data is done
[16:05:27] <hashar>	 I will trigger the full rep
[16:05:28] <mutante>	 root@gerrit2002:/srv/gerrit/data/lfs# du -hs .
[16:05:28] <mutante>	 20G	.
[16:05:43] <mutante>	 hashar: cool, yes please
[16:05:59] <mutante>	 owner of lfs files is gerrit2:gerrit2
[16:06:29] <mutante>	 /srv/gerrit/plugins still exists but is empty
[16:06:41] <mutante>	 we can merge that one change that stops managing it.. at some time
[16:06:57] <hashar>	 Caused by: org.apache.sshd.common.SshException: DefaultAuthFuture[ssh-connection]: Failed (UnsupportedCredentialItem) to execute: ssh://gerrit2@gerrit2002.wikimedia.org:22: org.eclipse.jgit.transport.CredentialItem$YesNoType:Accept this key and continue connecting all the same?
[16:06:58] <hashar>	 :)
[16:07:09] <icinga-wm>	 PROBLEM - gerrit process on gerrit2002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[16:07:37] <hashar>	 I thought we had the host key managed by puppet
[16:07:57] <icinga-wm>	 ACKNOWLEDGEMENT - Check systemd state on gerrit2002 is CRITICAL: CRITICAL - degraded: The following units failed: gerrit.service daniel_zahn https://phabricator.wikimedia.org/T334521 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[16:07:57] <icinga-wm>	 ACKNOWLEDGEMENT - gerrit process on gerrit2002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site daniel_zahn https://phabricator.wikimedia.org/T334521 https://wikitech.wikimedia.org/wiki/Gerrit
[16:08:03] <mutante>	 hashar: same
[16:08:30] <mutante>	 hashar: at least it's not "this is an RSA key, refuse to use it" or something? heh
[16:08:38] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[16:08:56] <mutante>	 yes, I also totally remember seeing this as managed in puppet
[16:09:07] <mutante>	 or got confused again about which key is which
[16:09:19] <hashar>	 at least with the ssh cli I can ssh :)
[16:09:26] <mutante>	 ok
[16:09:28] <hashar>	 maybe the primary needs to be restarted cause the host key is cached
[16:09:36] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[16:10:00] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[16:10:03] <mutante>	 hashar: maybe it needed puppet runs first on the other machine!
[16:10:30] <mutante>	 or what you said
[16:11:02] <hashar>	 I am restarting the primary gerrit
[16:11:08] <mutante>	 ok
[16:11:43] <mutante>	 manifests/init.pp:    String                            $ssh_host_key      = 'ssh_host_key',
[16:11:54] <mutante>	 manifests/init.pp:        file { "${gerrit_site}/etc/ssh_host_key":
[16:12:02] <hashar>	 the thing I dont get is the known_hosts on gerrit1003 is dated from May 17th
[16:12:05] <mutante>	 manifests/replication_key.pp:class gerrit::replication_key(
[16:12:27] <hashar>	 I am assuming that is when it got updated
[16:13:00] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[16:13:12] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[16:13:25] <mutante>	 on May 17th is when I merged changes to remove gerrit1001 from known_hosts and ssh config
[16:13:39] <hashar>	 ah
[16:13:41] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10Traffic, and 3 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[16:13:59] <mutante>	 also when I merged "add gerrit1003"
[16:14:28] <hashar>	 * 65e7228c75f - Update the known host key for gerrit2002.wikimedia.org (10 months ago) <Ahmon Dancy>| 
[16:14:33] <mutante>	 eh, no, that was https://gerrit.wikimedia.org/r/c/operations/puppet/+/919402 
[16:14:38] <mutante>	 but still, it's gotta be in there
[16:14:48] <hashar>	 and with the reimaging the key of the ssh daemon would have changed
[16:15:00] <mutante>	 right
[16:15:35] <hashar>	 then i can ssh as gerrit2 user from gerrit1003:/
[16:15:47] <mutante>	 hold on . it's like https://gerrit.wikimedia.org/r/c/operations/puppet/+/919405  but for the new host
[16:16:00] <hashar>	 OH i know
[16:16:05] <hashar>	 ssh uses the global known hosts
[16:16:25] <hashar>	 while Gerrit only relies on /var/lib/gerrit2/review_site/.ssh/known_hosts which is provided by puppet
[16:16:29] <hashar>	 and out of date
[16:17:02] <mutante>	 patch coming up
[16:17:32] <hashar>	 clearly that file needs to be maintained by Puppet :]
[16:18:04] <dancy>	 I have a vague recollection of suggesting that /var/lib/gerrit2/review_site/.ssh/known_hosts be a symlink to the global known hosts file. 
[16:18:51] <mutante>	 please review https://gerrit.wikimedia.org/r/c/operations/puppet/+/923378
[16:18:52] <hashar>	 yeah that would work, we would just need to add the github one to it
[16:18:54] <mutante>	 :)
[16:19:03] <hashar>	 or maybe Gerrit / Jgit supports multiple known host files
[16:19:17] <hashar>	 OpenSSH has support for both global and user files
[16:19:20] <hashar>	 grep gerrit2002 /etc/ssh/ssh_known_hosts  >> .ssh/known_hosts
[16:19:31] <icinga-wm>	 RECOVERY - gerrit process on gerrit2002 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit
[16:19:32] <hashar>	 I did that as a hack (and removed the preivous entry) and it replicated
[16:19:34] <mutante>	 this file is already managed by puppet
[16:19:37] <hashar>	 so no need to restart gerrit on the primary
[16:19:48] <mutante>	 if you can take a look at the link above you will see 
[16:20:16] <mutante>	 if you look manually at /etc/ssh/ssh_host_rsa_key.pub on gerrit2002 now that it's reimaged
[16:20:22] <mutante>	 and can confirm that is what I am adding here..
[16:20:37] <hashar>	 +1
[16:20:41] <mutante>	 :)
[16:21:30] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, 10Patch-For-Review, and 3 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Func) It seems the `wmf_deploy` branch should be us...
[16:22:12] <mutante>	 Info: Computing checksum on file /var/lib/gerrit2/.ssh/known_hosts
[16:22:31] <mutante>	 [/var/lib/gerrit2/.ssh/known_hosts]/content: content changed
[16:22:33] <mutante>	 try it now
[16:23:02] <hashar>	 I am filing a task
[16:23:12] <hashar>	 and yes it is doing the replication
[16:23:30] <mutante>	 isnt it simply "forgot to update the key after reimage"
[16:23:54] <mutante>	 just forgot to have this step in the list
[16:24:18] <hashar>	 so we already did that back in July 2022 https://phabricator.wikimedia.org/T313553  :]
[16:24:36] <mutante>	 ah
[16:24:56] <mutante>	 but meanwhile it's not as bad anymore
[16:24:56] <hashar>	 and in my exploration I found jgit learned `GlobalKnownHostsFile` and `UserKnownHostsFile`
[16:25:02] <mutante>	 the "manually crafted known_hosts " isnt' manual anymore
[16:25:04] <mutante>	 it's in puppet
[16:25:06] <hashar>	 so the jgit config can use those to point to the /etc/ssh/ssh_known_hosts
[16:25:18] <hashar>	 and an extra one that is specific to gerrit which only has the github one
[16:25:31] <dancy>	 Perfect
[16:25:58] <mutante>	 is everything working now that puppet change the known_hosts after you manually edited it
[16:26:02] <mutante>	 just making sure
[16:26:11] <brennen>	 (when absolutely nothing else is going on, i'd like to do a php-fpm restart on phab1004 to comment out a single bogus debug log statement that makes debugging anything there quite a bit noisier.)
[16:26:20] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap, 10Patch-For-Review: Handle spurious failures in Scap's self-installer - https://phabricator.wikimedia.org/T337394 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/146  install_local_version.sh: retry failed ven...
[16:26:31] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap, 10Patch-For-Review: Handle spurious failures in Scap's self-installer - https://phabricator.wikimedia.org/T337394 (10CodeReviewBot)
[16:26:43] <mutante>	 brennen: gerrit replication is going on but I think it's sufficiently unrelatet to phab things :)
[16:26:57] <brennen>	 kk, going ahead.
[16:27:38] <brennen>	 !log phab1004: sudo systemctl restart php7.3-fpm.service && sudo systemctl restart phd
[16:27:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:29:06] <mutante>	 for the record, when gerrit-replica.wm.org is down..as in gerrit is stopped.. it shows the nice maintenance page.. but when the service is up it will show a 404 but that will be status quo when opening it in browser
[16:29:19] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10PyBal, 10SRE, and 4 others: High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage - https://phabricator.wikimedia.org/T337497 (10jcrespo)
[16:31:44] <wikibugs>	 10Gerrit: Gerrit ssh replication should use the global ssh known_hosts - https://phabricator.wikimedia.org/T313553 (10hashar) This has hit us when we reimaged gerrit2002 since the ssh host key changed.  Potentially we can adjust the Puppet file `modules/gerrit/files/site/etc/jgit.config` to have:  * `GlobalKnown...
[16:32:04] <hashar>	 mutante: yes the replica does not offer a web service
[16:32:57] <mutante>	 yep, just pointing out that it's normal that it looks kind of like no data
[16:33:04] <mutante>	 in browser
[16:37:12] <hashar>	 I am triggering the full replication
[16:37:41] <hashar>	 !log ssh -p 29418 gerrit.wikimedia.org replication start --url gerrit2002 --all --wait  # T334521
[16:37:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:37:44] <stashbot>	 T334521: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521
[16:38:11] <hashar>	 ssh -p 29418 gerrit.wikimedia.org replication list --detail |wc -l
[16:38:11] <hashar>	 2767
[16:39:01] <mutante>	 hashar: is gerrit service supposed to run now?
[16:39:02] <hashar>	 and that can be checked by tailing on gerrit1003: /var/log/gerrit/replication_log
[16:39:20] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Clean up Phabricator production error logs - https://phabricator.wikimedia.org/T337500 (10brennen)
[16:39:21] <mutante>	 cool, ACK
[16:39:23] <hashar>	 Replicate All-Projects ref ..all.. to gerrit2002.wikimedia.org, Succeeded! (OK)
[16:39:23] <hashar>	 Replication of All-Projects ref ..all.. completed to 1 nodes, 
[16:39:23] <hashar>	 Replicate All-Users ref ..all.. to gerrit2002.wikimedia.org, Succeeded! (OK)
[16:39:23] <hashar>	 Replication of All-Users ref ..all.. completed to 1 nodes, 
[16:39:28] <hashar>	 apparently the meta repo got replicated
[16:39:51] <mutante>	 did you unmask it at some point?
[16:39:53] <hashar>	 which should be enough for the gerrit daemon to start
[16:39:56] <hashar>	 doing now
[16:40:06] <mutante>	 because it's already up and running again
[16:40:11] <claime>	 hashar: you cool with accepting the request for semver access to the gh org form sbasset?
[16:40:42] <hashar>	 claime: I have no idea :] nobody owns it per se
[16:40:49] <mutante>	 I am folliwng the replication_log too now
[16:40:57] <hashar>	 the only thing that we more or less manage is the replication of gerrit repo to github
[16:41:11] <hashar>	 but I don't think we have any policy regarding adding apps to github :/
[16:41:12] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10User-brennen: Clean up Phabricator production error logs - https://phabricator.wikimedia.org/T337500 (10CodeReviewBot) brennen opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/8  ManiphestGetT...
[16:41:17] <hashar>	 so I don't have an answer
[16:41:21] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10User-brennen: Clean up Phabricator production error logs - https://phabricator.wikimedia.org/T337500 (10CodeReviewBot)
[16:41:44] <mutante>	 the whole "wikimedia org on github" thing is "organically grown" and noone knows 
[16:41:54] * claime hangs head
[16:41:56] <claime>	 ok
[16:41:57] <hashar>	 ah fun times
[16:42:30] <mutante>	 probably the best that can be done is first compile the list of current admins there
[16:42:33] <hashar>	 so gerrit2002, yeah it needs some indices
[16:42:44] <mutante>	 and then go through the list and check who is even active
[16:43:12] <mutante>	 maybe security team feels like doing that audit
[16:43:46] <claime>	 Yeah, I maybe won't make them do that audit before giving them access
[16:43:48] <mutante>	 hashar: wanna confirm the lfs data is there? I copied 20G
[16:44:12] <claime>	 "Audit yourself who can give you perms to add an app to github plz"
[16:44:20] <hashar>	 !log gerrit2002: creating lucene indices: `java -jar /var/lib/gerrit2/review_site/bin/gerrit.war reindex --index groups` #  T334521
[16:44:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:44:22] <stashbot>	 T334521: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521
[16:45:16] <hashar>	 gerrit is up and runnig
[16:45:46] <hashar>	 Replicate mediawiki/core ref ..all.. to gerrit2002.wikimedia.org, Succeeded! (OK)
[16:45:47] <hashar>	 :]
[16:46:07] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10valerio.bozzolan) >>! In T333885#8875041, @brennen wrote: > https://phab.wmflabs.org/...
[16:48:10] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Scap, 10Patch-For-Review: Handle spurious failures in Scap's self-installer - https://phabricator.wikimedia.org/T337394 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/146  install_local_version.sh: retry failed ven...
[16:50:06] <wikibugs>	 10Gerrit: Gerrit daemon on a new replica refuses to start due to lack of `groups` index - https://phabricator.wikimedia.org/T337502 (10hashar)
[16:50:40] <hashar>	 mutante: so yeah it is working and slowly replicating the repositories over 4 threads
[16:51:14] <mutante>	 hashar: alright
[16:51:21] <mutante>	 I see the moving log
[16:51:57] <hashar>	 looks like it is busy creating the 2000 is repos :)
[16:55:43] <hashar>	 now I wanna check the cpu/IO load a full replication causes to both hosts
[16:55:50] <hashar>	 and maybe later we can bump the number of threads
[16:58:11] <hashar>	 https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=gerrit2002&var-datasource=thanos&var-cluster=misc
[16:58:16] <hashar>	 well it is barely doing anything
[16:58:42] <hashar>	 and looking at git receiving prrocesses on gerrit2002 there are constantly two and sometime 3
[16:58:49] <hashar>	 so I guess there is a lot of overhead between the transfers
[16:59:02] <hashar>	 and we should be able to bump the number of replication threads without troubles
[16:59:07] <wikibugs>	 (03PS1) 10Jforrester: function-*: Update links for source and coverage reports [integration/docroot] - 10https://gerrit.wikimedia.org/r/923394 (https://phabricator.wikimedia.org/T337425)
[17:02:44] <mutante>	 sounds good
[17:07:23] <hashar>	 Google doc lets me insert emoji but there is no org chart / tree representation grrr
[17:15:19] <mutante>	 hashar: /srv/gerrit/git on gerrit2002 is a bit larger than /srv/gerrit/git on gerrit1003
[17:21:01] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Fundraising Tech - Chaos Crew, 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, and 5 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10greg)
[17:25:04] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10Dzahn)
[17:25:31] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10Dzahn) gerrit2002 has been reimaged and is back up and running on bullseye, currently replication from gerrit1003 is ongoing.
[17:28:44] <mutante>	 hashar: I am thinking about copying the data off of gerrit1001 to gerrit1003 to a special place because then I can have it both.. you still have the data but I can decom the physical machine 
[17:28:58] <hashar>	 please hold on that
[17:29:03] <mutante>	 why?
[17:29:29] <hashar>	 I wanna dig into them, there is probably nothing that needs salvaging
[17:29:41] <hashar>	 given the legacy git repos and home dir got copied  on gerrit1003
[17:29:46] <mutante>	 yea, but you can totally dig into them this way
[17:29:52] <hashar>	 so there is most probably no need to make yet another copy of everything
[17:30:28] <mutante>	 yea, but "make aanother copy" isn't a problem and keeping a buster machine alive is
[17:30:56] <hashar>	 !log gerrit2002 replication eventually has completed at some point: `Replication completed with some errors!` # T334521
[17:30:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:30:58] <stashbot>	 T334521: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521
[17:31:20] <mutante>	 we have lots of space on /srv/ still
[17:31:20] <hashar>	 well leave it around for a few days :]
[17:31:34] <hashar>	 I am pretty sure there is nothing that needs to be kept there
[17:35:42] <mutante>	 ok.. though.... if we would just make a copy we could have it both ways.. not sure what the issue with that is
[17:36:20] <thcipriani>	 what's the timeline to decomm the old instances?
[17:36:26] <thcipriani>	 (completely)
[17:37:11] <mutante>	 tried to ask 
[17:37:30] <wikibugs>	 10Project-Admins: Create a #openstack-magnum - https://phabricator.wikimedia.org/T336941 (10rook)
[17:37:45] <thcipriani>	 does serviceops have any kind of hard deadline?
[17:40:24] <thcipriani>	 asking since it may factor in to whether or not hashar wants a copy; i.e., if there's no time to investigate in-place, let's make a copy and investigate at our own pace.
[17:41:10] <mutante>	 I would say either end of quarter or July 17th
[17:41:45] <mutante>	 besides that it's just going to bug me that I cant call it resolved and it's stalled at 99% of the goal
[17:42:10] <thcipriani>	 that's fair, so you'd like to shut down the new servers as soon as possible
[17:42:28] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508 (10hashar)
[17:42:43] <thcipriani>	 but if we have to, we can keep them until June 30 (min(end-of-quarter, July 17))
[17:42:45] <mutante>	 I don't have an actual problem waiting a little bit longer.
[17:43:12] <mutante>	 I also know though how likely things are to take longer
[17:43:30] <mutante>	 and it just be nice to be able to focus on the next thing
[17:43:59] <thcipriani>	 hashar: how much time do you need to investigate anything to keep on the decomm'd gerrit machines?
[17:46:23] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10hashar) The full replication log which I triggered from my machine is P48560: It ends with: `counterexample Replication completed with some er...
[17:46:48] <mutante>	 "funny refname"
[17:48:39] <mutante>	 at least it's not too many?
[17:48:54] <mutante>	 https://gerrit-replica.wikimedia.org/r/  is like before
[17:49:42] <thcipriani>	 funny. refname.
[17:49:46] <thcipriani>	 like haha funny?
[17:50:12] <hashar>	 yeah jgit doesn't like pushing `refs/master`
[17:50:17] <hashar>	 it is going to be a rabbit hole for sure
[17:50:38] <hashar>	 more concerning is I can't git clone from the replica. The packfile is delivered but somehow locally it can't checkout
[17:50:47] <hashar>	 git clone https://gerrit-replica.wikimedia.org/r/test/gerrit-ping
[17:50:50] <mutante>	 seems like it "contains characters ..  a special meaning in a Git object reference expression"
[17:51:01] <mutante>	 or "dangerous characters"
[17:52:02] <mutante>	 hashar: git clone worked for me
[17:52:17] <hashar>	 it is lfs related
[17:52:24] <dancy>	 Cloning hangs for me at the end... yeah.. lfs related
[17:52:36] <hashar>	 strace is such a useful tool
[17:52:41] <thcipriani>	 oh right...I have a vague memory of refs/master vs refs/heads/master. Same with these others. refs/forx
[17:53:50] <mutante>	 so, lfs.config has directory = /srv/gerrit/data/lfs  and that is where the data was copied to
[17:53:50] <hashar>	 feel free to cookie leak T334521 :]
[17:53:51] <stashbot>	 T334521: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521
[17:54:08] <mutante>	 though it was copied from gerrit1003 while gerrit1003 had running gerrit
[17:54:28] <mutante>	 do we need to resync the lfs data? I can 
[17:55:35] <wmf-insecte>	 maintenance-disconnect-full-disks build 494280 integration-agent-docker-1037 (/: 29%, /srv: 99%, /var/lib/docker: 49%): OFFLINE due to disk space
[17:55:52] <hashar>	 mutante: na it is unrelated
[17:55:59] <mutante>	 ok
[17:56:16] <hashar>	 I think that is because there is no web service on the replica and thus lfs can't fetch the objects from the end point
[17:56:18] <hashar>	 something like that
[17:56:22] <hashar>	 I don't think it is a concern
[17:57:06] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508 (10thcipriani) Superficially similar to {T275946}
[17:57:36] <hashar>	 the workaround is to use the ssh protocol for git AND LFS:
[17:57:38] <hashar>	 git clone ssh://hashar@gerrit-replica.wikimedia.org:29418/test/gerrit-ping
[17:57:52] <hashar>	 so yeah working
[17:58:09] <mutante>	 ok
[18:00:33] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[18:00:39] <hashar>	 mutante: I wrote a summary at https://phabricator.wikimedia.org/T334521#8881027
[18:00:40] <wmf-insecte>	 maintenance-disconnect-full-disks build 494281 integration-agent-docker-1037 (/: 29%, /srv: 19%, /var/lib/docker: 46%): RECOVERY disk space OK
[18:00:57] <hashar>	 if monitoring has been reenabled and is all green, I think you can finally mark this one resolved! congrats
[18:01:02] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab, 10Patch-For-Review: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10hashar) gerrit2002.wikimedia.org seems to be up and operational the primary successfully replicates to it :]  I think that concludes our adven...
[18:02:19] <hashar>	 I am off for dinner, poke me on IRC if needed, I have it on my phone :]
[18:04:34] <mutante>	 hashar: thanks:) sounds good. well, I can't, because of gerrit1001, but let's chat another day
[18:04:46] <mutante>	 good for today:)
[18:05:32] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[18:05:58] <hashar>	 mutante: yeah I promise I will look at what is leftover on gerrit1001 :]
[18:06:00] <mutante>	 enjoy dinner, also getting food. interesting that this resoled now ^
[18:06:08] <mutante>	 alright, ty
[18:17:13] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508 (10hashar) I found that code following the jGit code path to push.  The `components` variable is initialized to `1` it is incremented for each `/` found after `refs/`....
[18:42:35] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508 (10thcipriani) >>! In T337508#8881101, @hashar wrote: > I found that code following the jGit code path to push.  The `components` variable is initialized to `1` it is...
[19:05:47] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10nshahquinn-wmf) >>! In T706#8879020, @Aklapper wrote: > @nshahquinn-wmf: Hi, basically [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) - if...
[19:23:56] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Radar), 10User-brennen: GitLab MR reviews throw 500 errors when commenting on some specific lines - https://phabricator.wikimedia.org/T337514 (10brennen)
[19:24:50] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen), 10Upstream: GitLab: Make public the default fork option - https://phabricator.wikimedia.org/T324016 (10brennen)
[19:37:05] <hashar>	 brennen: want me to merge the `phplog()` removal or do you explicitly want Chad to do it?  (re: https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/8 )
[19:39:23] <brennen>	 hashar: go for it!
[19:39:41] <brennen>	 i just had to pick... somebody to review, and there's only one assignment available per MR...
[19:39:47] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review, 10User-brennen: Clean up Phabricator production error logs - https://phabricator.wikimedia.org/T337500 (10CodeReviewBot) hashar merged https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/8  ManiphestGetTa...
[19:39:51] <brennen>	 thanks
[19:39:59] <hashar>	 DONE! :D
[20:00:27] <wikibugs>	 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10Esanders) Hmm - that might be an improvement.  On a separate note, how often does it poll the API, or is it just when a new patchset is uploaded? If it's just when a new patchset...
[20:12:24] <wikibugs>	 10Gerrit, 10Patch-For-Review: [wm-checks-api] Create a new gerrit bot for Patch Demo - https://phabricator.wikimedia.org/T332474 (10hashar) The plugin runs 100% from the client browser, so anytime one load the change or refresh it due to a new patchset, the plugin will refresh the results. Additionally it fetc...
[20:12:25] <wikibugs>	 (03PS1) 10Hashar: wm-patch-demo: use WARNING to prevent chipset collapsing [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/923418 (https://phabricator.wikimedia.org/T332474)
[20:17:24] <wikibugs>	 10Project-Admins: Create a project for the WMF Movement Insights team - https://phabricator.wikimedia.org/T337518 (10nshahquinn-wmf)
[20:38:13] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Fundraising Tech - Chaos Crew, 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, and 5 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10thcipriani) >>! In T337427#8879118, @L...
[20:39:53] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Fundraising Tech - Chaos Crew, 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, and 5 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Ladsgroup) It's in the wmf_deploy bran...
[20:42:41] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Fundraising Tech - Chaos Crew, 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, and 5 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10thcipriani) We branch CentralNotice ea...
[21:41:38] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Fundraising Tech - Chaos Crew, 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, and 5 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10thcipriani) hrm, seems like the backpo...
[21:51:52] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Fundraising Tech - Chaos Crew, 10MediaWiki-Language-converter, 10MediaWiki-extensions-CentralNotice, and 5 others: LanguageConverter: Call to member function replace() on null - https://phabricator.wikimedia.org/T337427 (10Umherirrender) >>! In T337427#8881790,...
[22:22:24] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Release, 10Train Deployments: 1.41.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T337525 (10thcipriani) p:05Triage→03Medium a:03dduvall
[22:27:42] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Release, 10Train Deployments: 1.41.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T337526 (10thcipriani) p:05Triage→03Medium a:03jeena
[22:31:54] <wikibugs>	 10Release-Engineering-Team (They Live 🕶️🧟), 10Release, 10Train Deployments: 1.41.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T337527 (10thcipriani) p:05Triage→03Medium a:03jnuche