[12:36:26] duesen poked me about https://gerrit.wikimedia.org/r/c/integration/quibble/+/760657 to stop defining `MW_INSTALL_PATH` [12:42:26] which I guess requires a new version to be tagged [12:47:53] (03PS1) 10Hashar: release: Quibble 1.4.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) [12:49:01] (03PS1) 10Hashar: release: Start 1.42 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 [12:51:42] kostajh: if you got some cycle to +2 a new release ;) [12:52:02] that is to unblock platform engineering team and stop defining MW_INSTALL_PATH [12:52:13] and I guess I will pair with duesen for the aftermath [12:52:19] (03CR) 10Kosta Harlan: [C: 03+2] release: Start 1.42 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 (owner: 10Hashar) [12:52:36] and the parent change :]] [12:52:49] (03CR) 10Kosta Harlan: [C: 03+2] release: Quibble 1.4.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) (owner: 10Hashar) [12:52:56] hashar: LGTM :) [12:52:56] <3 [12:53:06] yeah I was just +2'ing the easy part first :P [12:53:06] I am sure that is going to break something somewhere [12:53:08] we will figure out [12:53:11] :D [12:53:21] yeah, possibly. or the changes in 1.4.0 which are not rolled out yet [12:53:50] oh [12:54:01] cause I was in vacations last week bah [12:54:24] it should be fine though... [12:54:54] (03PS2) 10Hashar: release: Start 1.4.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 [12:55:04] commit message had a typo 1.42 > 1.4.2 [12:55:37] ==> quibble-buster/changelog <== [12:55:37] quibble-buster (1.4.1) wikimedia; urgency=high [12:55:39] ::-((( [13:08:13] version 1.42... predictions for what calendar year that will be released? [13:15:27] (03CR) 10jerkins-bot: [V: 04-1] release: Quibble 1.4.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) (owner: 10Hashar) [13:17:28] oh [13:17:34] well really I should give up on that [13:17:43] cause I am again going to spend half a day fixing all the breakage :( [13:18:28] or those API tests are flappy :-\ [13:19:08] hashar: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/761320/9#message-26eda0f3f116d321edf6e3e27fcea68c271e5f2b it should be straightforward to fix [13:19:28] Daniel re-enabled some tests, but there is a config flag that needs to be set in Quibble for those to pass [13:19:51] I'll make a patch [13:20:00] so we need one more extra patch to quibble before 1.4.1? [13:20:16] yeah I think so. Or to disable the ci-fullrun-extensions job... [13:21:22] (03PS1) 10Kosta Harlan: LocalSettings: Use ForceDeferredUpdatesPreSend [integration/quibble] - 10https://gerrit.wikimedia.org/r/763223 (https://phabricator.wikimedia.org/T230211) [13:21:26] (03CR) 10jerkins-bot: [V: 04-1] release: Start 1.4.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 (owner: 10Hashar) [13:31:32] hashar: ^ for the patch https://gerrit.wikimedia.org/r/763223 [13:32:04] it's possible that actually causes lots of things to break. I guess we will find out [13:38:04] that is my worriness yes [13:38:12] so I think I will hold on that one [13:38:51] I gotta rename a docker image :/ [13:47:24] (03CR) 10jerkins-bot: [V: 04-1] LocalSettings: Use ForceDeferredUpdatesPreSend [integration/quibble] - 10https://gerrit.wikimedia.org/r/763223 (https://phabricator.wikimedia.org/T230211) (owner: 10Kosta Harlan) [13:47:59] another option is to disable those tests again in API testing [13:59:28] docker image renamed [13:59:58] what I understood is some test assume that jobs get run [14:00:10] but we have an helper that let tests force run the jobs before doing assertion [14:01:00] or we gotta disable those tests again since they still fail :-\ [14:38:11] 10Quibble, 10BlueSpice, 10Continuous-Integration-Config, 10ci-test-error: quibble-composer-mysql-php72-selenium-docker does not see service wiring from composer package - https://phabricator.wikimedia.org/T301066 (10Osnard) 05Open→03Resolved a:03Osnard Thanks for your investigation! This issue has n... [14:46:37] I think we should disable those tests again [14:47:04] It may be some weird thing with how the ci-fullrun-extensions environment is set up, I am not sure. [15:32:42] (03CR) 10Kosta Harlan: "This is a good patch to have IMO but doesn't solve the build failure. See https://gerrit.wikimedia.org/r/c/mediawiki/core/+/763262 for dis" [integration/quibble] - 10https://gerrit.wikimedia.org/r/763223 (https://phabricator.wikimedia.org/T230211) (owner: 10Kosta Harlan) [15:33:12] (03CR) 10Kosta Harlan: [C: 03+2] "Would be unblocked by I90bea3fda48efdc7e915908d00535df220c4cc69" [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) (owner: 10Hashar) [15:42:02] back [15:42:06] cause well stuff happened [15:45:23] * duesen wibbles [15:45:27] kostajh: I am chatting with Daniel about https://gerrit.wikimedia.org/r/c/mediawiki/core/+/763262/2/tests/api-testing/REST/ContributionsCount.js [15:45:31] :] [15:45:52] So https://gerrit.wikimedia.org/r/c/mediawiki/core/+/761320 didn't fix the flakyness of ContributionsCount.js ? [15:46:18] That's frustrating... we are bound to see similar issues with practically all API tests sooner or later. [15:46:26] apparently not cause the tests fail when we run quibble iwth a bunch of extensions ( https://gerrit.wikimedia.org/r/c/integration/quibble/+/763223 ) [15:46:35] Or we might. Depending on the root cause. Which apparently isn't what I thought it was. [15:46:44] the build failure is https://integration.wikimedia.org/ci/job/integration-quibble-fullrun-extensions/24/console [15:47:02] Does the test instance use a master/replica setup? [15:47:10] so that magically GET /user/{user}/contributions/count has the wrong count (it is off by 1) [15:47:26] supposedly cause the deferred job updating the counter hasn't run yet [15:47:43] supposedly. but if that was the case, my patch should have fixed it [15:47:44] it is a single primary MySQL which is spinned up by Quibble [15:48:42] OH [15:48:54] well that depends on where the fix got applied [15:49:17] merged into master yesterday. [15:49:22] hmm no, that build failed with Quibble installed from the branch [15:49:25] it re-enabled these tests as well. [15:51:43] So, if it's not replication lag, and it's not deferred job, then it has to be some cache somewhere. [15:52:12] or the new $wgForceDeferredUpdatesPreSend setting doesn't quite work? [15:52:39] no idea :-\ [15:52:53] One thing that confuses me is... user contribs come from the revision table. The revision table is updated in an atomic update before the edit returns. There is nothing deferred about it. [15:57:24] aren't `GET /user/{user}/contributions/count` returning a counter that is updated by `[DeferredUpdates] DeferredUpdates::run: started UserEditCountUpdate #XXX`? [15:57:37] cause I see a bunch of DeferredUpdates like that (2599 exactly) [15:58:06] then the debug log is 70MB of text which is a bit annoying to dig in :-\ [15:58:57] we should probably have the timestamp and reqid in the debug log lines [15:59:19] anyway, it seems the easier for now is to disable temporarily [16:03:53] (03CR) 10jerkins-bot: [V: 04-1] release: Quibble 1.4.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) (owner: 10Hashar) [16:03:55] (03CR) 10jerkins-bot: [V: 04-1] release: Start 1.4.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 (owner: 10Hashar) [16:03:57] (03CR) 10jerkins-bot: [V: 04-1] release: Start 1.4.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 (owner: 10Hashar) [16:04:08] That is what I thought originally. But no: contributions/count uses ContributionsLookup::getContributionCount() which does a COUNT(*) on the revision table. [16:04:18] I think it'S because the endpoint supports filtering by tage [16:04:46] We could actually optimize the common case of getting the total count, and use user_editcount for that. But currently, we do not, as far as i can tell [16:04:55] I summarized on https://phabricator.wikimedia.org/T301100#7715058 [16:05:28] Anyway, the problem with disabling temporarily is that it's not going to be temporary if it's not high on anyones list of priorities to fix it [16:06:00] I agree [16:06:01] I'd really like to understand what'S going on here :/ [16:06:18] though the flappy test hinder various other things such as quibble development unfortunately :-\ [16:06:38] yes, sure. this needs to be merged. [16:06:46] maybe that can be debugged with an adhoc log bucket [16:07:05] or working on enhancing the very dumb debug log (which is yet another adventure) [16:08:27] I don't even know what to log... [16:08:35] or what to look for [16:08:38] it's a mystery. [16:08:54] anyway, i gave +2 on Kosta's patch [16:10:02] those issues are indeed painful to investigate :-\ [17:09:29] (03CR) 10Kosta Harlan: release: Quibble 1.4.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) (owner: 10Hashar) [17:09:32] (03CR) 10Kosta Harlan: [C: 03+2] release: Quibble 1.4.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) (owner: 10Hashar) [17:09:43] (03CR) 10Kosta Harlan: [C: 03+2] release: Start 1.4.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 (owner: 10Hashar) [17:27:13] there is a new "Software Engineer in Test" role that we are hiring for, AIUI the idea is in part to help fix things like this. So, as long as there is a phab task and some good project tags, there is hope [17:46:03] the job integration-quibble-fullrun-extensions takes a while to run ;- [17:46:04] :( [17:46:47] I guess that is the price of running every single tests of every single repos participating :D [17:49:24] <<< Finish: npm test in /workspace/src/extensions/GrowthExperiments, in 150.811 s [17:49:25] :D [17:49:47] I am blaming eslint for being too slow [17:50:29] anyway I am off. see you tomorrow [17:57:44] (03Merged) 10jenkins-bot: release: Quibble 1.4.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/763217 (https://phabricator.wikimedia.org/T300301) (owner: 10Hashar) [17:58:30] (03Merged) 10jenkins-bot: release: Start 1.4.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/763218 (owner: 10Hashar) [18:11:41] going to tag that [18:21:13] Quibble 1.4.1 tagged [18:21:57] for the CI images that will have to wait tomorrow it is 7pm+ here ;-]]] [21:13:26] (03PS1) 10Kosta Harlan: [WIP] Logging: Use MWLoggerDefaultSpi [integration/quibble] - 10https://gerrit.wikimedia.org/r/763333 [21:19:26] (03PS2) 10Kosta Harlan: [WIP] Logging: Use MWLoggerDefaultSpi [integration/quibble] - 10https://gerrit.wikimedia.org/r/763333 [21:26:16] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Logging: Use MWLoggerDefaultSpi [integration/quibble] - 10https://gerrit.wikimedia.org/r/763333 (owner: 10Kosta Harlan) [21:26:32] (03PS3) 10Kosta Harlan: [WIP] Logging: Use MWLoggerDefaultSpi [integration/quibble] - 10https://gerrit.wikimedia.org/r/763333 [21:52:16] (03CR) 10Hashar: "<3" [integration/quibble] - 10https://gerrit.wikimedia.org/r/763333 (owner: 10Kosta Harlan)