[00:25:45] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:45:41] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:46:05] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [02:52:35] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [03:10:54] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.37.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T278347 (10Krinkle) [03:51:41] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [04:54:49] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [06:51:21] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Patch-For-Review, 10User-Addshore, 10User-zeljkofilipin: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) 3x the number of errors looking at all recorded logs today. Some big culprits are: ✖ "bef... [08:00:32] hashar: Hey! 2 questions pour toi: comment est-ce qu'on archive un repo dans gerrit (https://gerrit.wikimedia.org/r/admin/repos/search/xgboost). [08:01:02] Et y a-t-il une procédure pour renommer "master" vers "main" dans nos repos java? [08:03:05] gehel: bonjour, en général on créé une task pour #cleanup , il y a une liste des taches à réaliser [08:06:00] 10Release-Engineering-Team, 10Discovery-Search, 10Projects-Cleanup: Archive the search/xgboot repo in gerrit - https://phabricator.wikimedia.org/T281674 (10Gehel) [08:06:09] hashar: and here is the task ^ [08:06:11] thanks! [08:07:21] gehel: so yeah that provides the guide lines ;) [08:07:24] ah [08:07:25] sorry [08:08:22] 10Release-Engineering-Team, 10Discovery-Search, 10Projects-Cleanup: Archive the search/xgboot repo in gerrit - https://phabricator.wikimedia.org/T281674 (10hashar) [08:08:24] there is a template, so we can just apply it [08:08:43] for master > main branch renaming, you should be able to do it for the java repo [08:08:44] BUT [08:08:46] I did a quick search on wikitech, but could not find the template [08:09:12] the master name might be used in publishing documentation or the zuul job filters, so some extra care is needed there [08:09:51] https://wikitech.wikimedia.org/wiki/Gerrit does not seem to mention anything about archiving / deleting repos [08:10:04] the renaming itself is straightforward: head to the repo settings, branches, create a "main" branch that points to the commit of the "master" branch , update the HEAD branch to point to main, delete the master one [08:10:10] the devil is the aftermath / side effects [08:10:18] about the master->main rename, I'll create a task and I'll try to document the steps before breaking anything [08:10:43] yeah, I'm sure we have dependencies that we don't know about. But the Java repos should be fairly well isolated [08:10:48] for archiving repos, the process has grown organically mostly driven by various volunteers [08:11:11] I just head to phabricator.wikimedia.org/tag/cleanup then in the sidebar use the "Fill an archive request" [08:11:22] which comes preloaded with a template for the task with bunch of placeholders [08:11:29] got it! [08:11:32] it is mostly for mediawiki extensions & skins [08:11:40] so bunch of tasks do not apply to other repos [08:11:58] Oh, you already updated the task! Thanks! [08:15:30] will do it [08:16:20] gehel: so xgboost is gone for real right? I cant slash it from CI / and remove the custom Docker image ? [08:16:50] yep, we experimented with it, but too many issues. [08:17:08] not that I know what xgboost is about beside it is probably a java app :] [08:17:09] It make sense to archive it rather than delete it, so that we can restart if we ever need to. [08:17:48] and search/mjolnir has a comment refering to xgboost [08:17:48] the docker files for the images will be in git history, that's good enough [08:17:54] build_timeout: 10 # minutes, xgboost takes a while to compile T184754 [08:17:55] T184754: many search-mjolnir-tox-docker jobs in aborted state - https://phabricator.wikimedia.org/T184754 [08:18:59] let me double check with Erik when he is around [08:19:19] it is just a timeout [08:19:27] but maybe moljnir still relies on xgboost somehow [08:19:42] I'm not going to dig into the code, faster to just ask Erik [08:19:43] (03PS1) 10Hashar: Archive search/xgboost [integration/config] - 10https://gerrit.wikimedia.org/r/684276 (https://phabricator.wikimedia.org/T281674) [08:20:03] erik bernhardson isn't it ? [08:21:08] (03CR) 10Hashar: Archive search/xgboost (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/684276 (https://phabricator.wikimedia.org/T281674) (owner: 10Hashar) [08:22:06] 10Release-Engineering-Team, 10Discovery-Search, 10Projects-Cleanup, 10Patch-For-Review: Archive the search/xgboot repo in gerrit - https://phabricator.wikimedia.org/T281674 (10Gehel) @EBernhardson the search/mjolnir CI configuration has a comment about xgboost ("build_timeout: 10 # minutes, xgboost takes a... [08:22:18] we will see [08:22:57] 10Release-Engineering-Team, 10Discovery-Search, 10Projects-Cleanup, 10Patch-For-Review: Archive the search/xgboot repo in gerrit - https://phabricator.wikimedia.org/T281674 (10hashar) The JJB config for the search/mjolnir has a comment referring to search/xgboost so maybe it is still used. Though most pro... [08:27:15] gehel: oh and I have a java question for you. Assuming some .jar that got compiled with java8, would it work with java 11 somehow magically? [08:27:21] or does that needs a recompilation [08:27:29] the java source code is java 8 / java 11 compatible [08:28:36] backward compatibility is usually really good. There is really no reason that the bytecode should be incompatible. There is a very small chance that the code is using some standard libs that have been removed in Java 11, but very unlikely [08:29:43] that is what I thought ;] [08:29:58] will not prevent me from rebuilding everything with java 11 anyway since I have to write down the doc hehe [08:30:08] :) [08:34:48] 10Continuous-Integration-Infrastructure, 10wdwb-tech (wdwb-tech (small tasks)): Trigger Jenkins jobs on patchset uploaded by the user Luca Mauri - https://phabricator.wikimedia.org/T281675 (10Michael) [08:42:47] (03CR) 10Michael Große: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/684282 (https://phabricator.wikimedia.org/T281675) (owner: 10Michael Große) [08:58:07] gehel: oh and for the branch renaming, that will be done as part of the gitlab migration ( https://phabricator.wikimedia.org/T279612#7050354 ) [08:58:36] 10Project-Admins: Request Project (cfssl-pki) for pki tasks - https://phabricator.wikimedia.org/T281371 (10Aklapper) Sure, please see https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects for required info (name, description, type, repo if applicable) [08:59:03] 10Project-Admins, 10Jade, 10Machine-Learning-Team: Archive Jade project - https://phabricator.wikimedia.org/T281430 (10Aklapper) [08:59:05] 10Beta-Cluster-Infrastructure, 10Jade, 10User-Ladsgroup: Undeploy JADE extension from beta cluster - https://phabricator.wikimedia.org/T281418 (10Aklapper) [08:59:50] 10Project-Admins, 10Jade, 10Machine-Learning-Team: Archive Jade project - https://phabricator.wikimedia.org/T281430 (10Aklapper) For future reference, #projects-cleanup links to a checklist template to use for such tasks [09:13:31] hashar: so it does not make sense to try doing it ahead of time ? [09:23:36] probably not ;) [09:23:45] (03CR) 10WMDE-leszek: [C: 03+1] Add Luca Mauri to the list of users triggering V+2 voting checks [integration/config] - 10https://gerrit.wikimedia.org/r/684282 (https://phabricator.wikimedia.org/T281675) (owner: 10Michael Große) [10:34:40] (03PS2) 10Michael Große: Add Luca Mauri to the list of users triggering V+2 voting checks [integration/config] - 10https://gerrit.wikimedia.org/r/684282 (https://phabricator.wikimedia.org/T281675) [10:36:15] (03CR) 10Michael Große: "(Updated to patchset 2 to use the correct comment syntax for yaml 🤦)" [integration/config] - 10https://gerrit.wikimedia.org/r/684282 (https://phabricator.wikimedia.org/T281675) (owner: 10Michael Große) [10:48:58] (03CR) 10Lars Wirzenius: [C: 03+2] feat: Fail fast if non-existent version is passed to scap prep [tools/scap] - 10https://gerrit.wikimedia.org/r/683943 (owner: 10Ahmon Dancy) [10:55:26] (03Merged) 10jenkins-bot: feat: Fail fast if non-existent version is passed to scap prep [tools/scap] - 10https://gerrit.wikimedia.org/r/683943 (owner: 10Ahmon Dancy) [10:55:56] (03PS1) 10Hashar: [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 [12:23:58] 10Beta-Cluster-Infrastructure, 10SRE-swift-storage: swift-ring: Add support for Cinder based Cloud VPS VMs - https://phabricator.wikimedia.org/T281699 (10Majavah) [13:39:42] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Readers-Web-Backlog (Kanbanana-FY-2020-21): MinervaNeue legacy ruby browser tests need to be updated to work on buster/ruby2.5 - https://phabricator.wikimedia.org/T280901 (10ovasileva) [13:59:49] (03PS1) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [14:12:25] (03PS2) 10Hashar: [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 [14:12:27] (03PS2) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [14:17:36] (03PS3) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [14:22:30] (03PS1) 10Hashar: Add job to build our Gerrit plugins [integration/config] - 10https://gerrit.wikimedia.org/r/684417 [14:38:21] TIL about 'chronic', thx liw [14:39:41] http://manpages.ubuntu.com/manpages/xenial/man1/chronic.1.html [14:51:56] 10Release-Engineering-Team (Doing), 10Phatality, 10Developer Productivity: Phatality: Move Request ID back to task description - https://phabricator.wikimedia.org/T237683 (10Krinkle) 05Open→03Resolved [15:11:12] 10Phabricator (Upstream), 10Legalpad, 10Upstream: Granularize Legalpad view/edit/sign permissions - https://phabricator.wikimedia.org/T194148 (10Aklapper) [15:13:49] (03PS3) 10Ahmon Dancy: Convert train-dev to use docker instead of VMs [tools/train-dev] - 10https://gerrit.wikimedia.org/r/684040 [15:14:26] 10Release-Engineering-Team, 10Projects-Cleanup, 10Discovery-Search (Current work), 10Patch-For-Review: Archive the search/xgboot repo in gerrit - https://phabricator.wikimedia.org/T281674 (10MPhamWMF) [15:21:35] Krinkle, yw [16:03:25] (03PS3) 10Jforrester: Zuul: Add Luca Mauri to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/684282 (https://phabricator.wikimedia.org/T281675) (owner: 10Michael Große) [16:03:34] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Luca Mauri to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/684282 (https://phabricator.wikimedia.org/T281675) (owner: 10Michael Große) [16:06:45] (03Merged) 10jenkins-bot: Zuul: Add Luca Mauri to the CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/684282 (https://phabricator.wikimedia.org/T281675) (owner: 10Michael Große) [16:07:06] !log Zuul: Add Luca Mauri to the CI allow list [16:07:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:08:23] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10User-Michael, 10wdwb-tech (wdwb-tech (small tasks)): Trigger Jenkins jobs on patchset uploaded by the user Luca Mauri - https://phabricator.wikimedia.org/T281675 (10Jdforrester-WMF) 05Open→03Resolved a:03Michael Deployed. [16:14:29] (03CR) 10Jforrester: [C: 03+2] Allow configuring prefixes in PrefixedGlobalFunctionsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661937 (https://phabricator.wikimedia.org/T274007) (owner: 10DannyS712) [16:18:47] (03Merged) 10jenkins-bot: Allow configuring prefixes in PrefixedGlobalFunctionsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/661937 (https://phabricator.wikimedia.org/T274007) (owner: 10DannyS712) [16:19:13] hashar: Hey, don't know if you've spotted, but I've got into a stuck situation where I need a new release of quibble to re-build the quibble stretch images. :-( [16:20:07] (03PS1) 10Hashar: [WMF] Add XDG_CACHE_HOME to tools/download_file.py [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684440 [16:20:54] James_F: nop sorry I have barely done any review today :\ [16:21:42] hashar: We need https://gerrit.wikimedia.org/r/c/integration/quibble/+/676452 which you wrote. Fun. :-( [16:22:35] hmm [16:22:40] that worked and now it fails [16:23:09] Yeah, because I upgraded the base image for that job. [16:23:20] It passed when I tested the previous run. [16:23:25] But now everything has broken. [16:23:29] * James_F sighs at himself. [16:24:29] the tox job tests quibble against python 3.5 / 3.7 (at least) [16:24:34] 3.5 is kept for Stretch images [16:24:39] which I guess we will drop evenutally [16:24:46] Very soon. [16:24:48] the fullrun job is well.. an integration job :] [16:25:00] and it is now running Buster and thus i guess python 3.7 [16:25:04] The only uses for stretch right now are for PHP 70/71 and for the ruby selenium jobs for Wikibase and MobileFrontend. [16:25:05] so something in the setup is broken [16:25:08] ModuleNotFoundError: No module named 'distutils.core' [16:25:13] guess that no more exists somehow [16:25:19] Yeah. [16:25:31] I'll have a poke later today. [16:27:25] (03CR) 10Hashar: "With the buster image:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/676452 (owner: 10Hashar) [16:27:29] TLDR: install the python3-distutils in the releng/quibble-buster image, its sole use would be to able to install Quibble :) [16:27:31] James_F: ^ ;] [16:27:49] I think that will do it [16:28:50] Ack. [16:28:52] Will do. [16:30:40] it is a bit of a pity [16:30:57] or maybe we can add distutils to the install_requires and have setuptools to install it for us [16:31:14] saves one from having to install the debian package / rebuild the images [16:31:24] (03PS1) 10Mholloway: Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 [16:31:44] (03Abandoned) 10Mholloway: Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 (owner: 10Mholloway) [16:32:10] I'm trying the rebuild locally now. [16:33:06] 10Continuous-Integration-Infrastructure, 10User-Michael, 10wdwb-tech (wdwb-tech (small tasks)): Trigger Jenkins jobs on patchset uploaded by the user Luca Mauri - https://phabricator.wikimedia.org/T281675 (10Michael) >>! In T281675#7053724, @Jdforrester-WMF wrote: > Deployed. Thank you 🙏 [16:33:29] James_F: I am off, gotta take some fresh air before the curfew :D [16:33:39] (03CR) 10Jforrester: dockerfiles: Update quibble-fresnel to use Fresnel 1.0.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/684004 (https://phabricator.wikimedia.org/T281543) (owner: 10Krinkle) [16:33:44] See you! [16:33:45] Evening; perhaps you already know but I couldn't see a ticket about it: the number of waiting gearman jobs seems to keep growing since 1800-ish UTC yesterday. See https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&from=now-24h&to=now I'm not even sure if this is a problem really hence why I stuck my head in here. anecdotally some people here at WMDE are claiming more jobs are timing out than usual. [16:35:32] oh men [16:36:38] $ zuul-gearman.py status|sort -k2 -g |tail -n2 [16:36:38] stop:contint2001.wikimedia.org 391 1 1 [16:36:38] set_description:contint2001.wikimedia.org 3365 0 1 [16:36:43] tarrow: yeah that is a bug somehow :( [16:37:04] hashar: sorry to tell you now :/ [16:37:17] well better than being called in the middle of the night! :]]] [16:37:37] i have absolutely no idea what is going, but that looks like an issue with Jenkins [16:37:39] go for freedom since it's been like this 24 hrs? [16:38:07] another hour won't hurt but you'll get to breathe some air :) [16:39:25] (03Restored) 10Mholloway: Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 (owner: 10Mholloway) [16:40:56] (03PS2) 10Mholloway: Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 [16:41:58] (03PS3) 10Mholloway: Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 (https://phabricator.wikimedia.org/T279089) [16:46:52] I suspect the Gearman plugin lacks ability to cancel a job or set the job descriptions [16:46:53] bah [16:47:02] which would be related to the recent upgrade of Jenkins [16:47:09] but I don't get why it would only happens now [16:47:44] (03PS2) 10Hashar: Add job to build our Gerrit plugins [integration/config] - 10https://gerrit.wikimedia.org/r/684417 [16:47:46] (03PS1) 10Hashar: dockerfiles: add maven settings to gerrit image [integration/config] - 10https://gerrit.wikimedia.org/r/684445 [16:48:13] (03CR) 10Ottomata: [C: 03+1] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 (https://phabricator.wikimedia.org/T279089) (owner: 10Mholloway) [16:52:28] tarrow: filed as https://phabricator.wikimedia.org/T281737 [16:52:29] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Zuul: Zuul can't stop jobs or set the build description - https://phabricator.wikimedia.org/T281737 (10hashar) [16:52:46] I have no idea why it pills up since Sunday 18:00 [16:53:11] hashar: thanks! sorry to be the evening drop of bad news [16:53:12] (03CR) 10Mholloway: [C: 03+2] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 (https://phabricator.wikimedia.org/T279089) (owner: 10Mholloway) [16:53:26] tarrow: thank you to have reported it ! :] [16:53:28] (03CR) 10Mholloway: [V: 03+2 C: 03+2] Review access change [schemas/event/secondary] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/684452 (https://phabricator.wikimedia.org/T279089) (owner: 10Mholloway) [16:55:09] 10Phabricator: Phabricator poor image UX - https://phabricator.wikimedia.org/T281738 (10Jidanni) [16:56:16] 10Phabricator: Phabricator poor image UX - https://phabricator.wikimedia.org/T281738 (10Jidanni) [16:57:41] 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10jbond) >>! In T276148#7050178, @Sergey.Trofimovsky.SF wrote: > Here it is, requesting settings review: > > https://ger... [16:58:34] (03PS1) 10Zfilipin: WIP Create selenium-daily-beta-AdvancedSearch [integration/config] - 10https://gerrit.wikimedia.org/r/684469 (https://phabricator.wikimedia.org/T267360) [17:00:22] (03PS1) 10Jforrester: dockerfiles: [quibble-buster] Add python3-distutils so quibble can build [integration/config] - 10https://gerrit.wikimedia.org/r/684470 [17:00:34] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [quibble-buster] Add python3-distutils so quibble can build [integration/config] - 10https://gerrit.wikimedia.org/r/684470 (owner: 10Jforrester) [17:04:42] (03Merged) 10jenkins-bot: dockerfiles: [quibble-buster] Add python3-distutils so quibble can build [integration/config] - 10https://gerrit.wikimedia.org/r/684470 (owner: 10Jforrester) [17:05:12] !log Docker: Publishing quibble-buster images with python3-distutils so quibble can build [17:05:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:05:17] 10Phabricator: Phabricator poor image UX - https://phabricator.wikimedia.org/T281738 (10Jidanni) Also when viewing the image, in the corner these have no mouseover saying what they do. Maybe the middle one means make a comment on the image. {F34439148} [17:07:15] (03CR) 10Lars Wirzenius: "Is this ready for merge? If not, should it be marked as work in progress?" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/684040 (owner: 10Ahmon Dancy) [17:11:07] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Zuul: Zuul can't stop jobs or set the build description - https://phabricator.wikimedia.org/T281737 (10hashar) p:05Triage→03Unbreak! a:03hashar The stop* and set_description* functions are handled by ManagementWorkerThr... [17:11:22] java.lang.StackOverflowError [17:11:22] at org.acegisecurity.context.SecurityContext$1.getAuthentication(SecurityContext.java:46) [17:11:23] at org.acegisecurity.context.SecurityContext$2.getAuthentication(SecurityContext.java:60) [17:11:23] at org.acegisecurity.context.SecurityContext$1.getAuthentication(SecurityContext.java:46) [17:11:24] haaaaaa [17:14:11] (03PS4) 10Ahmon Dancy: WIP: Convert train-dev to use docker instead of VMs [tools/train-dev] - 10https://gerrit.wikimedia.org/r/684040 [17:20:18] restarted CI Jenkins [17:27:59] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Zuul: Zuul can't stop jobs or set the build description - https://phabricator.wikimedia.org/T281737 (10hashar) p:05Unbreak!→03Medium After restarting, the manager is: ` $ zuul-gearman.py workers|cut -b-90|grep manager 21... [17:28:25] so I had to restart Jenkins cause the Gearman management thread died out somehow (probably related to the recent Jenkins upgrade) [17:28:37] and it registered the management worker thread under a different name [17:28:47] so there is a pill up of functions in Gearman which can't really be flushed [17:29:07] unless we restart Jenkins until the worker thread manages to connect as contint2001 instead of the ip [17:29:18] it is not a big deal to loose the stop and set_description backlog [17:29:26] but the graph is wrong [17:29:41] and I think it can only be cleared out by restarting the Zuul scheduler [17:30:01] I guess I will restart it tomorrow when CI is less busy [17:31:35] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Zuul: Zuul can't stop jobs or set the build description - https://phabricator.wikimedia.org/T281737 (10hashar) The graph will remain high until the Gearman server is restarted to flush the backlog. That has to be done by rest... [17:31:36] I am off [17:31:41] CI is working at least [17:33:26] 10Phabricator: Phabricator poor image UX - https://phabricator.wikimedia.org/T281738 (10Aklapper) 05Open→03Invalid See https://www.mediawiki.org/wiki/How_to_report_a_bug which says "Include only one problem per task" [17:37:02] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Zuul: Zuul can't stop jobs or set the build description - https://phabricator.wikimedia.org/T281737 (10hashar) The code: ` name=src/main/java/hudson/plugins/gearman/GearmanProxy.java // constructor private GearmanProx... [17:49:41] (03CR) 10Xqt: Add project dashboard to Pywikibot Gerrit (031 comment) [pywikibot/core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/614034 (owner: 10Dvorapa) [17:58:20] (03CR) 10Dvorapa: Add project dashboard to Pywikibot Gerrit (031 comment) [pywikibot/core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/614034 (owner: 10Dvorapa) [18:18:13] (03PS3) 10Jack Phoenix: Add dependencies for social tools and some other things that I maintain [integration/config] - 10https://gerrit.wikimedia.org/r/684455 [18:19:50] (03CR) 10jerkins-bot: [V: 04-1] Add dependencies for social tools and some other things that I maintain [integration/config] - 10https://gerrit.wikimedia.org/r/684455 (owner: 10Jack Phoenix) [18:27:22] (03PS1) 10Jforrester: jjb: [integration-quibble-fullrun] Use image with python3-distutils [integration/config] - 10https://gerrit.wikimedia.org/r/684483 [18:27:24] (03PS1) 10Jforrester: jjb: [integration-quibble-apache-fullrun] Use image with python3-distutils [integration/config] - 10https://gerrit.wikimedia.org/r/684484 [18:27:37] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/684483 (owner: 10Jforrester) [18:27:42] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/684484 (owner: 10Jforrester) [18:33:05] (03Merged) 10jenkins-bot: jjb: [integration-quibble-fullrun] Use image with python3-distutils [integration/config] - 10https://gerrit.wikimedia.org/r/684483 (owner: 10Jforrester) [18:34:36] (03Merged) 10jenkins-bot: jjb: [integration-quibble-apache-fullrun] Use image with python3-distutils [integration/config] - 10https://gerrit.wikimedia.org/r/684484 (owner: 10Jforrester) [18:35:25] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MediaWiki-ResourceLoader, 10Performance-Team, 10Developer Productivity: Run ForeignResourceManager verification on MediaWiki core commits - https://phabricator.wikimedia.org/T203694 (10Krinkle) [18:35:29] (03CR) 10Xqt: Add project dashboard to Pywikibot Gerrit (031 comment) [pywikibot/core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/614034 (owner: 10Dvorapa) [18:37:37] (03CR) 10Jforrester: [C: 03+2] "CI should now be fixed." [integration/quibble] - 10https://gerrit.wikimedia.org/r/676452 (owner: 10Hashar) [18:37:48] (03CR) 10Jforrester: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/676405 (owner: 10Hashar) [18:38:44] (03CR) 10Jforrester: [C: 03+2] "check experimental" [integration/quibble] - 10https://gerrit.wikimedia.org/r/676452 (owner: 10Hashar) [18:56:12] (03Merged) 10jenkins-bot: setuptools_scm 6.0.0 no more support python 3.5 [integration/quibble] - 10https://gerrit.wikimedia.org/r/676452 (owner: 10Hashar) [18:57:45] (03PS4) 10Jforrester: CoreNpmComposerTest: Run `composer test-some` with paths [integration/quibble] - 10https://gerrit.wikimedia.org/r/663917 (https://phabricator.wikimedia.org/T199403) [18:59:16] (03CR) 10Dvorapa: Add project dashboard to Pywikibot Gerrit (031 comment) [pywikibot/core] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/614034 (owner: 10Dvorapa) [19:11:41] 10LibUp: libraryupgrader missed some repos due to Generic.Files.LineLength.TooLong - https://phabricator.wikimedia.org/T203830 (10Legoktm) 05Declined→03Open p:05Triage→03Lowest I would like to properly fix this at some point. [19:33:34] (03PS1) 10Nikki Nikkhoui: Add publish step for image-suggestion-api [integration/config] - 10https://gerrit.wikimedia.org/r/684493 (https://phabricator.wikimedia.org/T281256) [19:36:33] (03PS2) 10Jforrester: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge [integration/config] - 10https://gerrit.wikimedia.org/r/684493 (https://phabricator.wikimedia.org/T281256) (owner: 10Nikki Nikkhoui) [19:36:38] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge [integration/config] - 10https://gerrit.wikimedia.org/r/684493 (https://phabricator.wikimedia.org/T281256) (owner: 10Nikki Nikkhoui) [19:38:46] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge [integration/config] - 10https://gerrit.wikimedia.org/r/684493 (https://phabricator.wikimedia.org/T281256) (owner: 10Nikki Nikkhoui) [19:42:31] !log Zuul: [mediawiki/services/image-suggestion-api] Publish images post-merge T281256 [19:42:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:42:34] T281256: Integrate container creation into the pipeline - https://phabricator.wikimedia.org/T281256 [20:49:52] 10Continuous-Integration-Config, 10Better Use Of Data, 10Product-Data-Infrastructure: Add CI jobs for mediawiki/libs/metrics-platform - https://phabricator.wikimedia.org/T279180 (10Mholloway) [21:11:40] https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&from=now-7d&to=now [21:11:44] something looks wrong with gearman [21:12:51] oh, nvm [21:13:00] 10:29:26 <+hashar> but the graph is wrong [21:13:23] legoktm: https://phabricator.wikimedia.org/P15673 [21:13:27] bah, you beat me making a paste [21:13:42] yeah, I'm reading https://phabricator.wikimedia.org/T281737 now [21:44:27] (03PS4) 10Jack Phoenix: Add dependencies for social tools and some other things that I maintain [integration/config] - 10https://gerrit.wikimedia.org/r/684455 [21:46:16] (03CR) 10jerkins-bot: [V: 04-1] Add dependencies for social tools and some other things that I maintain [integration/config] - 10https://gerrit.wikimedia.org/r/684455 (owner: 10Jack Phoenix) [22:05:41] (03PS5) 10Jack Phoenix: Add dependencies for social tools and some other things that I maintain [integration/config] - 10https://gerrit.wikimedia.org/r/684455 [22:07:04] (03CR) 10jerkins-bot: [V: 04-1] Add dependencies for social tools and some other things that I maintain [integration/config] - 10https://gerrit.wikimedia.org/r/684455 (owner: 10Jack Phoenix) [22:43:04] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Readers-Web-Backlog (Kanbanana-FY-2020-21): MinervaNeue legacy ruby browser tests need to be updated to work on buster/ruby2.5 - https://phabricator.wikimedia.org/T280901 (10Jdlrobson) a:05ovasileva→03None [22:43:16] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Readers-Web-Backlog (Kanbanana-FY-2020-21): MinervaNeue legacy ruby browser tests need to be updated to work on buster/ruby2.5 - https://phabricator.wikimedia.org/T280901 (10Jdlrobson) Team should discuss (and maybe estimate) what our plan is here. Do... [23:09:57] I'm trying to figure out why a patch that passed CI last week is failing this week - with no other obvious changes. It's now hitting a setup issue. I see a few changes were merged since then to config - although none strike me as related. - are there any changes that could impact fundraising CI? https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/5099/console vs [23:09:57] https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/5099/console [23:11:17] that's the same link twice :) [23:11:46] 00:01:47 PHP Fatal error: Uncaught Symfony\Component\DependencyInjection\Exception\ServiceCircularReferenceException: Circular reference detected for service "db.mysql_precreated", path: "db.mysql_precreated -> instances". in phar:///src/wikimedia/fundraising/crm/civicrm-buildkit/bin/amp/vendor/symfony/dependency-injection/Symfony/Component/DependencyInjection/Container.php:296 [23:11:57] Is it installing/using the same versions of the depedancies? [23:14:38] sorry the one that worked is https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/5086/console [23:14:44] & yeah they should be the same [23:15:14] that whole db.mysql_precreated is stuff we want to unravel (it was necessary under vagrant) but I don't know why it stopped working [23:19:29] oh I think I've found something [23:19:45] in one of our repos [23:41:31] 10Release-Engineering-Team, 10Projects-Cleanup, 10Discovery-Search (Current work), 10Patch-For-Review: Archive the search/xgboot repo in gerrit - https://phabricator.wikimedia.org/T281674 (10EBernhardson) > Though most probably it is an artifact from the past and the code has since been updated to no more... [23:45:45] (03CR) 10Ebernhardson: [C: 03+1] Archive search/xgboost (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/684276 (https://phabricator.wikimedia.org/T281674) (owner: 10Hashar)