[00:21:01] Project beta-update-databases-eqiad build #45085: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/45085/ [01:21:00] Project beta-update-databases-eqiad build #45086: 04STILL FAILING in 59 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/45086/ [01:29:22] Yippee, build fixed! [01:29:22] Project beta-update-databases-eqiad build #45087: 09FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/45087/ [04:33:45] (03PS7) 10Jeena Huneidi: Refactor promote step to use withCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/628234 [04:34:48] (03CR) 10Jeena Huneidi: "patchset created when testing locally: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/628985" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/628234 (owner: 10Jeena Huneidi) [04:38:51] (03CR) 10Jeena Huneidi: Refactor promote step to use withCredentials (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/628234 (owner: 10Jeena Huneidi) [04:40:07] (03PS8) 10Jeena Huneidi: Refactor promote step to use withCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/628234 [04:42:12] (03CR) 10Jeena Huneidi: Refactor promote step to use withCredentials (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/628234 (owner: 10Jeena Huneidi) [07:35:27] 10phan-taint-check-plugin, 10Platform Team Initiatives (Parsoid REST API in PHP (CDP2)): phan-taint-check-plugin: Undefined constant 'ast\AST_LIST' - https://phabricator.wikimedia.org/T263500 (10hashar) There are two root causes: ---- On Monday I have upgraded #quibble on CI with a logic change has to when... [07:35:47] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10phan-taint-check-plugin, 10Quibble, 10Platform Team Initiatives (Parsoid REST API in PHP (CDP2)): phan-taint-check-plugin: Undefined constant '... - https://phabricator.wikimedia.org/T263500 [08:01:50] (03PS1) 10Hashar: tests: plan for mediawiki/services/parsoid [integration/quibble] - 10https://gerrit.wikimedia.org/r/629057 (https://phabricator.wikimedia.org/T263500) [08:02:55] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10phan-taint-check-plugin, 10Quibble, and 2 others: phan-taint-check-plugin: Undefined constant 'ast\AST_LIST' - https://phabricator.wikimedia.org/T263500 (10hashar) TLDR, the new Quibble 0.... [08:03:12] (03CR) 10jerkins-bot: [V: 04-1] tests: plan for mediawiki/services/parsoid [integration/quibble] - 10https://gerrit.wikimedia.org/r/629057 (https://phabricator.wikimedia.org/T263500) (owner: 10Hashar) [09:04:15] (03CR) 10Hashar: Implement webserver engines as subclasses (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/628759 (https://phabricator.wikimedia.org/T225218) (owner: 10Awight) [09:10:05] (03CR) 10Awight: Implement webserver engines as subclasses (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/628759 (https://phabricator.wikimedia.org/T225218) (owner: 10Awight) [09:22:14] (03CR) 10Hashar: [C: 04-1] "That is double casting to int() cause it was already done above:" (033 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/628181 (owner: 10Thcipriani) [09:31:22] hashar: https://integration.wikimedia.org/ci/job/mwgate-node10-docker/163159/console does this warrant a phab task? ("npm WARN tar ENOSPC: no space left on device, write") [09:35:28] hashar: thanks for your patches to unbreak phan in CI, I'm taking a look. We're going to have a problem because of T254571 though [09:35:29] T254571: Teach LibUp how to migrate to new phan using ConfigBuilder - https://phabricator.wikimedia.org/T254571 [09:48:44] Daimona: ah so maybe ConfigBuilder should bump the phan config minor version (0.11.0 instead of 0.10.3) [09:49:03] kostajh: yeah looks like there is an infra issue of some sort :-\ [09:49:31] Yeah I even wanted to make it 1.0.0 at first [09:50:10] I left a comment on gerrit; the problem is, the ConfigBuilder was merged before the taint-check upgrade, so I think it's not trivial to tag a release that includes the latter but not the former, right? [09:52:30] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @lsobanski: I've added you. //Usual default disclaimer: Please follow [guidelines]( https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Creating_new_p... [10:05:03] Daimona: I commented about ideas to branch it https://gerrit.wikimedia.org/r/c/mediawiki/tools/phan/+/629060 [10:05:16] Daimona: I am out for lunch then meeting. Wil lbe back around in roughly 1 hour and a half [10:05:46] kostajh: I think that is the npm/composer caches filing the partition when too many jobs run in parallel [10:06:02] kostajh: the agents have a rather smallish partition and our caches keep growing [10:06:20] ah [10:09:00] I am off be back for meeting and all after lunch [10:54:31] (03PS3) 10Awight: Implement webserver engines as subclasses [integration/quibble] - 10https://gerrit.wikimedia.org/r/628759 (https://phabricator.wikimedia.org/T225218) [10:54:33] (03PS18) 10Awight: [WIP] Add option for using Apache as server [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [10:54:35] (03PS1) 10Awight: Explicit registry for backends [integration/quibble] - 10https://gerrit.wikimedia.org/r/629086 [10:58:31] (03CR) 10Hashar: [C: 03+1] Explicit registry for backends [integration/quibble] - 10https://gerrit.wikimedia.org/r/629086 (owner: 10Awight) [11:02:00] (03CR) 10Hashar: [C: 03+2] "Nicer :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/628759 (https://phabricator.wikimedia.org/T225218) (owner: 10Awight) [11:17:10] (03PS2) 10Awight: Fix service repo tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/629057 (https://phabricator.wikimedia.org/T263500) (owner: 10Hashar) [11:51:52] awight: got something working (mostly) with the releng/dev-images configs. I'll push a separate patch, I think? [13:14:40] 10Continuous-Integration-Infrastructure, 10User-Addshore: un blacklist https://integration.wikimedia.org/ci/computer/XXXX/builds - https://phabricator.wikimedia.org/T178458 (10CDanis) @Kormat has just tripped over this issue as well, trying to view her own past builds. At this point wouldn't it be better to t... [13:16:44] 10Continuous-Integration-Infrastructure, 10User-Addshore: Remove DENY rules for https://integration.wikimedia.org/ci/computer/XXXX/builds - https://phabricator.wikimedia.org/T178458 (10Addshore) [13:33:03] Daimona: I have send the hotfix for phan :] [13:37:45] * Daimona looking [13:39:35] turned out to be straightforward [13:39:53] Daimona: should the ConfigBuilder thing be evetually upstreamed to esty/phan directly? [13:39:58] seems that could benefit everyone [13:40:34] So, the plan looks good and I have approved it. Let me recheck git log locally [13:40:55] I merely copy pasted from git log --oneline [13:41:03] and moved the couple important changes at the top [13:41:10] Well, phan already has a similar builder, although much more powerful, different semantics, etc. [13:41:14] rests are all build/test/development related [13:41:44] I actually plan to make this ConfigBuilder MW-specific, by providing convenience methods to add extensions, filter out vendor folders, etc. [13:42:43] ahhh cool ;] [13:44:45] Alright, the git log matches what we expect [13:48:09] BTW, why is this failing? https://integration.wikimedia.org/ci/job/mw-tools-phan-mwcore-testrun/63/console [13:48:56] Ah, I guess it's the "Archiving artifacts" step [13:49:34] Nevermind, I'm going to approve the release. Shall I create the tag? [13:57:57] (03PS3) 10Hashar: Only run linters for extensions/skins [integration/quibble] - 10https://gerrit.wikimedia.org/r/629057 (https://phabricator.wikimedia.org/T263500) [13:58:42] (03CR) 10Hashar: "Repurposed changing "not is_core" by "(is_extension or is_skin)" which rules out parsoid :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/629057 (https://phabricator.wikimedia.org/T263500) (owner: 10Hashar) [13:58:55] Daimona: yeah sure! [13:59:04] Daimona: I am looking at the build failure [13:59:56] maybe it does not generate anything under /log [14:01:32] the container runs: vendor/bin/phan --long-progress-bar --require-config-exists [14:01:47] and the Jenkins job archives antyhing written to log but would fail if nothing is there [14:02:04] if we get Phan to write something there, that would fix it [14:05:27] Or can we make it not expect any write? [14:06:00] I thought about generating a checkstyle report [14:06:08] -m checkstyle -o checkstyle.xml [14:07:11] I guess that would work, yes. [14:14:39] (03PS1) 10Hashar: dockerfiles: pass mediawiki-phan-testrun args to phan [integration/config] - 10https://gerrit.wikimedia.org/r/629131 [14:14:41] (03PS1) 10Hashar: Add checkstyle output for phan testrun [integration/config] - 10https://gerrit.wikimedia.org/r/629132 [14:14:44] Daimona: the magic is ^^ :] [14:16:01] (03CR) 10Hashar: [C: 03+2] "Building building" [integration/config] - 10https://gerrit.wikimedia.org/r/629131 (owner: 10Hashar) [14:16:26] Hah, that seems an easy fix :D Thank you [14:17:24] I guess you can tag 0.10.3 now? [14:17:43] (03Merged) 10jenkins-bot: dockerfiles: pass mediawiki-phan-testrun args to phan [integration/config] - 10https://gerrit.wikimedia.org/r/629131 (owner: 10Hashar) [14:18:04] !log Building Docker image releng/mediawiki-phan-testrun:0.2.0 [14:18:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:20:43] Sure, I'll tag [14:21:01] Unrelated, is there anything I could do to move forward with https://gerrit.wikimedia.org/r/c/integration/config/+/567938 ? [14:21:37] wasn’t there a request recently-ish to add significant changes or possible problems to the train blocker tasks ahead of time? [14:21:45] I can’t find it now :S [14:21:59] (I could just write a free-form comment but I think there was a template to use somewhere) [14:24:04] Daimona: https://integration.wikimedia.org/ci/job/mw-tools-phan-mwcore-testrun/64/console success! :) [14:24:19] Daimona: and the build page shows the checkstyle report https://integration.wikimedia.org/ci/job/mw-tools-phan-mwcore-testrun/64/ :] [14:24:41] (03CR) 10Hashar: "Job updated and it works!" [integration/config] - 10https://gerrit.wikimedia.org/r/629132 (owner: 10Hashar) [14:24:45] (03CR) 10Hashar: [C: 03+2] Add checkstyle output for phan testrun [integration/config] - 10https://gerrit.wikimedia.org/r/629132 (owner: 10Hashar) [14:25:03] R [14:25:18] Very good, thank you! Also, 0.10.3 tagged: https://gerrit.wikimedia.org/g/mediawiki/tools/phan/+/refs/tags/0.10.3 [14:25:56] (03Merged) 10jenkins-bot: Add checkstyle output for phan testrun [integration/config] - 10https://gerrit.wikimedia.org/r/629132 (owner: 10Hashar) [14:27:05] 10Gerrit: Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" - https://phabricator.wikimedia.org/T263293 (10matmarex) Fine by me, but it looks like @xSavitar was having the same problem? [14:27:22] Daimona: awesome [14:27:33] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10phan-taint-check-plugin, 10Quibble, and 2 others: phan-taint-check-plugin: Undefined constant 'ast\AST_LIST' - https://phabricator.wikimedia.org/T263500 (10hashar) **status** The workarou... [14:27:34] and I have proposed the fix in parsoid: https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/629135/ [14:27:39] hopefully packagist will catch up fast [14:27:46] I am off [14:27:53] have a meeting tonight. Thank you veyr much Daimona ! [14:28:21] Thank you :-) [14:29:22] * hashar vanishes [14:30:04] I’m now going backwards through train blocker tasks to see if I can find this callout / potential issues / heads-up template [14:30:17] surely I didn’t hallucinate it? [14:30:55] aha!! “risky” is the word to search for (e. g. T256668#6281786) [14:30:56] T256668: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 [14:31:43] https://wikitech.wikimedia.org/wiki/Deployments/Risky_change_template [14:31:55] * Lucas_WMDE bookmarks that link with a million synonyms for future reference [14:36:26] 10Release-Engineering-Team: Add link to risky change template to deployment blockers tasks - https://phabricator.wikimedia.org/T263547 (10Lucas_Werkmeister_WMDE) [14:38:06] 10Gerrit: Can't `git pull` mediawiki/core from Gerrit: "fatal: the remote end hung up unexpectedly" - https://phabricator.wikimedia.org/T263293 (10xSavitar) >>! In T263293#6483706, @matmarex wrote: > Fine by me, but it looks like @xSavitar was having the same problem? Yes @matmarex. But sometimes it works too.... [14:44:36] Will we see a base image with support for Python 3.8 in the wmf docker registry anytime soon? I suspect after release of Debian Bullseye? [14:45:11] Saw there was some task in Phabricator for support in Jenkins or so. [15:15:01] marxarelli: before I venture into figuring this out for myself, do you know of any notes about using Blubber locally with a Docker Compose setup? Context is that I'm starting a new project that will eventually be deployed on the prod k8s cluster, but trying to keep the dev environment less complicated than running a local k8s cluster. [15:20:42] (03CR) 10Lars Wirzenius: [C: 03+2] Use /etc/wikimedia-cluster to choose the wikiversion file [tools/release] - 10https://gerrit.wikimedia.org/r/627907 (owner: 10Ahmon Dancy) [15:21:23] (03Merged) 10jenkins-bot: Use /etc/wikimedia-cluster to choose the wikiversion file [tools/release] - 10https://gerrit.wikimedia.org/r/627907 (owner: 10Ahmon Dancy) [15:27:09] 10LibUp, 10phan, 10phan-taint-check-plugin: Release bugfix for ast\AST_LIST in phan-taint-check-plugin to unstuck libup updates on some repos - https://phabricator.wikimedia.org/T262451 (10Daimona) 05Open→03Resolved a:03hashar [16:27:26] bd808: i haven't any notes on that but i'm up for brainstorming [16:28:05] i'm not very experienced with docker-compose but seems like it assumes a Dockerfile is a specific location? [16:29:00] marxarelli: You can tell it to look in arbitrary places in modern versions. I'm starting to mess with it locally. I'll reach out when I know enough to ask good questions :) [16:29:41] green field projects are fun for experiments, but I have to keep my love of building tooling in check ;) [16:29:59] also relevant: newer versions of docker with buildkit enabled allow for a `#syntax=[buildkit backend image ref]` instruction at the beginning of the Dockerfile, and there's an experimental patch for blubber that adds support for such a buildkit backend [16:30:31] i hadn't sought merging of the patch because there wasn't a tangible use case for it, but if you can provide me one i will refresh it :) [16:30:54] bd808: haha, yeah i hear ya [16:30:58] marxarelli: ooh. you may have something to help me with this: do you know where I can look at a blubber config for a python app? [16:31:50] er... can't remember off the top of my head [16:32:53] 10MediaWiki-Releasing, 10Security: Tracking bug for MediaWiki 1.31.4/1.32.4/1.33.1 security release - https://phabricator.wikimedia.org/T225152 (10Reedy) [16:35:12] bd808: but yeah, basically, if docker compose can point to an arbitrary dockerfile, and we merge the support for buildkit to blubber, you could basically point it to a blubber.yaml instead that contains e.g. a `# syntax=docker-registry.wikimedia.org/blubber-buildkit` instruction, and... presto chango [16:35:35] now you're building with blubber [16:35:45] oh, that would be super cute [16:37:06] :) [16:37:24] https://docs.docker.com/compose/compose-file/#build is the spec on what you can tell docker-compose about how to build an image. Basically you can set the $PWD for the build and point to an arbitrary Dockerfile with build args [16:37:51] i want to get buildkit support into blubber at some point regardless since buildkit supports all kinds of good things [16:37:59] ah, nice [16:38:18] My first stab is actually just going to be adding a target to my makefile that will generate a Dockerfile using Blubber [16:38:42] sounds like a good start [16:39:27] fwiw, you can use blubberoid.wikimedia.org too [16:39:31] e.g. [16:39:35] https://www.irccloud.com/pastebin/gyEvzgpn/ [16:40:08] approximates the basic blubber cli using our production deployed blubberoid microservice [16:47:41] looks like joewalshwmf has played with some similar ideas (blubber + docker-compose) in https://github.com/wikimedia/ocean [16:58:45] oh neat [17:13:37] "/usr/bin/python3: No module named easy_install" -- I think that Blubber may not quite know how to work with python3 [17:14:53] Python 2 forever!!! [17:15:27] https://i.imgflip.com/4fvx4s.jpg - One Does Not Simply - "Easy" Install [17:17:32] easy_install is ancient stuff for bootstrapping setuptools, which itself is more and more an ancient tool [17:39:11] 10LibUp, 10phan, 10phan-taint-check-plugin: Release bugfix for ast\AST_LIST in phan-taint-check-plugin to unstuck libup updates on some repos - https://phabricator.wikimedia.org/T262451 (10Umherirrender) >>! In T262451#6483777, @gerritbot wrote: > Change 629141 **merged** by jenkins-bot: > [labs/libraryupgra... [17:54:40] bd808: patches welcome :) [17:55:42] you might be able to get away with using the generic `builder:` config [17:56:47] most of the language/package-manager specific stuff in blubber builds on the generic builder implementation anyhow which itself follows a simple pattern of 1) what are the files you need to run your build commnad; 2) what is your build command [17:57:54] but the node and python specific implementations also try to output package manager commands in such a way that will lead to a dockerfile with cacheable layers [18:01:19] (03CR) 1020after4: [C: 03+2] "This seems correct but we may have been relying on the incorrect behavior for a long time. I guess lets try it and see if anything breaks" [tools/scap] - 10https://gerrit.wikimedia.org/r/628170 (owner: 10Ahmon Dancy) [18:01:26] bd808: is https://docs.python.org/3/library/ensurepip.html a better approach for python3? [18:03:33] marxarelli: maybe... Is the intent of Blubber to ensure that any python in the container already can be bootstrapped into using pip? [18:04:05] basically yeah [18:05:48] I'll think on it a bit. Interestingly when I changed base images that error went away and I don't know why [18:20:46] (03CR) 10Dduvall: [C: 03+2] "Nice work; looks good!" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/628234 (owner: 10Jeena Huneidi) [18:21:34] (03Merged) 10jenkins-bot: Refactor promote step to use withCredentials [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/628234 (owner: 10Jeena Huneidi) [19:27:17] marxarelli: if I made a patch to blubber that made it possible to run builder steps in PhasePrivledged, would that ever have a chance of being merged? [19:27:37] * bd808 knows he is messing with the flow of the universe here [19:32:28] I would not merge it :) [19:33:11] you're no fun with your product integrity and such ;) [19:33:23] Haha [19:33:33] Fork it on gitlab... [19:35:18] If it will never mainline then that doesn't help me get closer to pipelinelib integration. :) [19:36:35] https://phabricator.wikimedia.org/T201875 might be of interest though [19:37:29] that would be nice! And I think it would let me do the goofy things I'm imaginging [19:38:20] I was already thinking about a custom command for the weirdness I want to try [19:39:56] What's the weirdness? [19:41:23] Right now I'm using poetry to manage my python app. Poetry is tool to manage and lock dependencies while also managing venvs for the code. [19:41:50] I have a manual Dockerfile that seems to work pretty well with it, but now I'm trying to get into the blubber/piplinelib world [19:42:17] so I need to either throw out all the awesome that poetry gives me, or smash poetry into a blubber built dockerfile [19:42:47] and the smashing hit a wall when I figured out that builder steps can't do any root things [19:43:16] I have an idea to work around it though... just more layers in the layering I think... [19:44:51] I see [19:45:40] Well the way we've approached things is to have folks out anything root in a base image managed by operations/production-images [19:46:05] the right thing to do is probably start working on a python/poetry builder, but I wanted a "quick" test since my golang is sketchy [19:46:54] So if you can get the poetry stuff into say a python-poetry image in that repo, you can put your unprivileged user stuff in blubber [19:47:40] ah, right. And I could POC that easily with a local base image [19:48:10] ways_to_skin_the_cat++ [19:48:21] Haha [19:49:20] Love that uint64 counter [19:49:40] So many ways... [20:09:43] (03PS1) 10Jeena Huneidi: pipeline config: promote to blubberoid chart [blubber] - 10https://gerrit.wikimedia.org/r/629211 [20:17:59] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Release, 10Train Deployments: 1.36.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T263177 (10thcipriani) p:05Triage→03Medium a:03mmodell [20:19:33] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T263178 (10thcipriani) p:05Triage→03Medium a:03hashar [20:20:53] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T263179 (10thcipriani) p:05Triage→03Medium a:03dduvall [20:21:05] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T263180 (10thcipriani) p:05Triage→03Medium a:03jeena [20:21:45] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T263181 (10thcipriani) p:05Triage→03Medium a:03LarsWirzenius [20:22:25] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T263182 (10thcipriani) p:05Triage→03Medium a:03brennen [20:23:03] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T263183 (10thcipriani) p:05Triage→03Medium a:03dancy [20:24:13] 10Release-Engineering-Team-TODO, 10Release, 10Train Deployments: 1.36.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T263184 (10thcipriani) p:05Triage→03Medium a:03hashar [20:29:09] thcipriani: I think gerrit-replica is having issues again, codesearch is seeing 502 errors [20:30:08] mut.ante is taking a look in -cloud [20:30:34] > java.lang.OutOfMemoryError [20:31:08] it was not running with the new parameters to create a dump if that happens [20:31:12] unlike gerrit1001 [20:31:16] restarting it [20:31:26] ah, bummer [20:31:47] and thanks [20:31:53] legoktm: try again? [20:32:00] i see a web UI again on the replica [20:32:38] looks to be working, thanks [20:33:19] https://gerrit-replica.wikimedia.org/r/monitoring?part=graph&graph=usedMemory&period=mois [20:33:48] now it's running with -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/srv/gerrit [20:33:52] wow, that server is serving more traffic than the primary [20:34:35] ah, no [20:34:45] misremembered graphs [20:35:05] order of magnitude was off [20:35:24] which, I guess, means that traffic has dropped off quite a bit on this server [20:35:30] https://gerrit-replica.wikimedia.org/r/monitoring?part=graph&graph=httpHitsRate&period=mois [20:36:40] hrm, maybe stuck threads here [20:36:45] pre-restart [20:36:53] would the new git 2.0 have helped lower traffic? [20:38:08] in this case it seems to coincide with a threadlock making fewer available threads https://gerrit-replica.wikimedia.org/r/monitoring?part=graph&graph=activeThreads (is one theory) [20:38:38] how can it be that {{CURRENTVERSION}} doesn't match what's shown on Special:Version? see https://test.wikipedia.org/wiki/Special:Version vs https://test.wikipedia.org/wiki/Main_Page [20:38:41] is codesearch using v2? [20:38:54] twentyafterfour: caching probably? [20:39:16] let me see [20:39:29] legoktm: that's what I thought but I also thought cache was disabled when you are logged in [20:39:52] er bypassed not disabled [20:39:57] only frontend/varnish caching [20:40:29] the sitenotice is stored in the message cache, which I don't think would invalidate automatically on a new version [20:41:09] ah ok [20:41:37] codesearch is running git 2.18.1 (through upstream's container), do we need to explicitly enable the 2.0 thing? [20:42:27] fwiw, the graph links are all Forbidden for me [20:42:58] mutante: you need to be logged in to the replica, https://gerrit-replica.wikimedia.org/r/login [20:43:54] ok, that works. just confusing because i was logged in on gerrit and the replica login also redirects me to main gerrit [20:44:16] hrm, I can't remember which version of git defaults to using protocol v2 [20:44:18] * thcipriani greps [20:46:12] https://www.infoq.com/news/2020/03/git-2-26-wire-protocol-2/ 2.26 [20:46:35] apparently it was introduced in 2.18 (https://www.infoq.com/news/2018/06/git-2.18-v2-protocol-commitgraph) [20:47:52] ah [20:48:14] yeah, git -c protocol.version=2 is what I used to play with it in older versions [20:49:10] should save quite a bit of overhead for repos with a lot of changes refs [20:49:25] ops/puppet has like 14MB of refs that come over when you run git fetch [20:51:17] hashar wrote a good post about it https://phabricator.wikimedia.org/phame/post/view/199/faster_source_code_fetches_thanks_to_git_protocol_version_2/ [20:52:14] gerrit-replica just needed restart. memory usage kept growing since months..slowly but steady [20:52:25] recently we restarted main gerrit but not the replica [20:52:45] maybe the replica even ran out first and then users switched to main gerrit causing the issue there [20:53:14] 10VPS-project-codesearch, 10git-protocol-v2: Have codesearch use git protocol v2 - https://phabricator.wikimedia.org/T263591 (10Legoktm) [20:56:54] [{exception_id}] {exception_url} ErrorException from line 3271 of /srv/mediawiki/php-1.36.0-wmf.10/languages/Language.php: Use of Language::commafy with a non-numeric string was deprecated in MediaWiki 1.36. [Called from Language::formatNum] [20:57:55] srsly? [20:58:14] 'commafy' heh [20:58:34] srsly. [20:58:42] https://github.com/wikimedia/mediawiki/commit/b9b8b536825eedd0c2f3de8fb5a18a90a3f58a2b [20:59:17] https://phabricator.wikimedia.org/T263592 [20:59:19] I guess that deprecation wants reverting, and changing for some sort of logging so we can see about fixing it [21:00:43] nice thanks reedy [21:01:00] * Reedy makes a patch to remove deprecation [21:01:05] Can cherry pick it across then at lesat [21:01:50] blerg. found a new way to stub my toe in blubber. Copies runs in install phase (which seems correct) but is placed in the generated Dockerfile *after* the builder step in the importing variant runs. This means my builder step can not use binaries copied in. [21:02:14] 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T257978 (10mmodell) [21:02:26] thcipriani: thanks, I'll poke you again once I'm done implementing it to see if it makes a noticable dent in traffic [21:03:12] legoktm: cool! sounds good. [21:09:00] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), 10phan-taint-check-plugin, 10Quibble, and 2 others: phan-taint-check-plugin: Undefined constant 'ast\AST_LIST' - https://phabricator.wikimedia.org/T263500 (10hashar) 05Open→03Stalled p:... [21:27:08] 10LibUp, 10phan, 10phan-taint-check-plugin: Release bugfix for ast\AST_LIST in phan-taint-check-plugin to unstuck libup updates on some repos - https://phabricator.wikimedia.org/T262451 (10hashar) I was not aware of this task. Daimona and I had a quick talk about it and we went with a `stable-0.10` branch w... [21:37:33] 10LibUp, 10git-protocol-v2: Have libup use git procotol v2 - https://phabricator.wikimedia.org/T263595 (10Legoktm) [22:05:20] 10LibUp, 10Patch-For-Review, 10git-protocol-v2: Have libup use git procotol v2 - https://phabricator.wikimedia.org/T263595 (10Legoktm) 05Open→03Resolved New container is running now. [22:41:09] 10VPS-project-codesearch, 10Patch-For-Review, 10git-protocol-v2: Have codesearch use git protocol v2 - https://phabricator.wikimedia.org/T263591 (10Legoktm) a:03Legoktm [22:43:42] 10LibUp, 10MediaWiki-Codesniffer, 10Patch-For-Review: Drop "php5,inc" from MediaWiki-CodeSniffer ruleset of extensions - https://phabricator.wikimedia.org/T200956 (10Legoktm) Example changes: * https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MassMessage/+/629232 * https://gerrit.wikimedia.org/r/c/media... [22:59:01] 10Release-Engineering-Team (Logspam), 10MediaWiki-Internationalization, 10Wikimedia-production-error: PHP Fatal error: Out of memory in cdb/src/Reader/DBA.php - https://phabricator.wikimedia.org/T260234 (10Krinkle) >>! In T247223#5960879, @Krinkle wrote:> > Reading out a localisation file with CDB takes up... [22:59:05] 10Release-Engineering-Team (Logspam), 10MediaWiki-Internationalization, 10Wikimedia-production-error: PHP Fatal error: Out of memory in cdb/src/Reader/DBA.php (via BlockManager.php) - https://phabricator.wikimedia.org/T260234 (10Krinkle) [22:59:12] 10Release-Engineering-Team (Logspam), 10MediaWiki-Internationalization, 10Wikimedia-production-error: PHP Fatal error: Out of memory in cdb/src/Reader/DBA.php (via BlockManager.php) - https://phabricator.wikimedia.org/T260234 (10Krinkle) p:05Triage→03Medium [23:10:03] 10Release-Engineering-Team (Logspam), 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10dancy) [23:10:55] 10Release-Engineering-Team (Logspam), 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10dancy) p:05Triage→03Low [23:12:08] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10Reedy) [23:12:42] 10Release-Engineering-Team (Logspam), 10GlobalUsage, 10MW-1.36-notes (1.36.0-wmf.10; 2020-09-22), 10Wikimedia-production-error: ApiQueryGlobalUsage.php Undefined index error when accessing $pageIds - https://phabricator.wikimedia.org/T263601 (10dancy) [23:31:25] 10VPS-project-codesearch, 10git-protocol-v2: Have codesearch use git protocol v2 - https://phabricator.wikimedia.org/T263591 (10Legoktm) 05Open→03Resolved A few tries later... ` legoktm@codesearch6:~$ cat /etc/hound-gitconfig # This file is managed by Puppet [protocol] version = 2 legoktm@codesearch6:~$ s... [23:31:25] thcipriani: both codesearch and libup should now be using git protocol v2 [23:31:37] that was fast :) [23:32:28] I'll be interested to see what effect that has, hopefully it means less load overall [23:33:24] :D I hope so