[00:12:32] 06Release-Engineering-Team, 06DBA, 06Editing-team, 06MediaWiki-Engineering, 07Wikimedia-production-error: [testwiki-wmf.20] Structured discussion (Flow) pages display Internal error - https://phabricator.wikimedia.org/T373831 (10Etonkovidova) 03NEW [00:14:24] 06Release-Engineering-Team, 06DBA, 06Editing-team, 06MediaWiki-Engineering, 07Wikimedia-production-error: [testwiki-wmf.20] Structured discussion (Flow) pages display Internal error - https://phabricator.wikimedia.org/T373831#10111465 (10Etonkovidova) [06:19:55] Project mediawiki-core-doxygen build #3944: 04FAILURE in 1 min 52 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/3944/ [07:28:36] Yippee, build fixed! [07:28:36] Project mediawiki-core-doxygen build #3945: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/3945/ [07:29:29] (03approved) 10jnuche: patchdemo: make and chown /var/www/.cache dir [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/65 (https://phabricator.wikimedia.org/T373721) (owner: 10jhuneidi) [08:12:16] "3: set: Illegal option -o pipefail" fun times :) [08:21:22] (03update) 10jnuche: jenkins-rel: do not set keepUndefinedParameters [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/73 (https://phabricator.wikimedia.org/T133737) (owner: 10hashar) [08:21:25] (03approved) 10jnuche: jenkins-rel: do not set keepUndefinedParameters [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/73 (https://phabricator.wikimedia.org/T133737) (owner: 10hashar) [08:23:20] (03update) 10hashar: Tweak flake8 config and lint make-release/automatic-branch-cut [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/109 [08:25:54] (03update) 10hashar: Tweak flake8 config and lint make-release/automatic-branch-cut [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/109 [08:31:54] jnuche: do you have any idea as to why the automatic branch cut job has failed with "set: Illegal option -o pipefail" ? [08:32:00] https://releases-jenkins.wikimedia.org/job/Automatic%20branch%20cut/254/console :) [08:32:55] hashar: yep, this is the cause: https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/commit/4ca4744163ae8d2bfd41cba952ddf3477549c926 [08:33:00] job is fixed now: https://releases-jenkins.wikimedia.org/job/Automatic%20branch%20cut/ [08:33:05] I'm currently re-running the presync [08:33:36] oh nice [08:34:02] that is I guess yet another issue I should flag for the Jenkins CI update which uses groovy as well ( integration/pipelinelib ) [08:34:02] fun [08:34:46] probably, if there's any Groovy code anywhere using that function [08:40:12] I don't get why the java upgrade would affect Groovy though :/ [08:44:15] I haven't looked into the Groovy code, but I can make an educated guess [08:44:38] Java 17 now bundles a `stripIndent` that has exactly the new behavior we just saw: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html#stripIndent() [08:44:48] that was missing in Java 11: https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/String.html [08:45:14] most likely the new Groovy implementation delegates that method to Java for version 17 [08:47:19] 10Continuous-Integration-Infrastructure, 07Jenkins: Verify Groovy behavior on CI Jenkins under Java 17 - https://phabricator.wikimedia.org/T373843 (10hashar) 03NEW [08:48:09] ahhhhh [08:48:26] that makes sense :) [08:49:17] make test [08:49:17] Makefile:17: *** recipe commences before first target. Stop. [08:49:18] whistles [08:52:05] (03PS1) 10Hashar: (DO NOT SUBMIT) are system tests running on CI? [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/1070204 [08:53:15] (03Abandoned) 10Hashar: (DO NOT SUBMIT) are system tests running on CI? [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/1070204 (owner: 10Hashar) [08:55:26] 10Continuous-Integration-Infrastructure, 07Jenkins: Verify Groovy behavior on CI Jenkins under Java 17 - https://phabricator.wikimedia.org/T373843#10111924 (10hashar) 05Open→03Resolved a:03hashar Pipelinelib has two usage of stripIndent() which seems already properly indented: ` lang=groovy,name=test... [10:55:27] cd 11 [11:01:18] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Sort out how to pull data (affiliations etc) from Bitergia DB via SortingHat API to find needed data updates - https://phabricator.wikimedia.org/T360762#10112437 (10Aklapper) 05Open→03Resolved * I finally updated public https://www... [11:34:10] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 4 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10112573 (10Lucas_Werkmeister_WMDE) >>! In T371633#10078899, @Krinkle wrote: > I'm going t... [11:36:29] jnuche: I have rebased the puppet change to add Java 17 to contint which dropped your CR+1 [11:36:46] I wanted to run a puppet compile and went to amend the commit message for the Hosts: header, that got rid of your vote [11:36:46] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1069325 :) [11:39:33] well, that's strange, the UI is still showing my vote everywhere, even after a disable the browser cache [11:39:44] e.g. [11:39:48] https://usercontent.irccloud-cdn.com/file/1J0TkcfI/image.png [11:41:13] oh man sorry :D [11:41:24] jnuche: I apologize! :D [11:48:27] !log Removing Maven Release plugin from CI Jenkins ( https://plugins.jenkins.io/m2release/ ). Was used until 2020 by analytics-refinery-release / wikidata-query-rdf-release-silent which instead went to use Docker images [11:48:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:57:52] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:02:52] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:03:22] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:08:22] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:13:28] ^ that is the observability infra having issues [12:36:09] (03PS1) 10Robert Vogel: Remove obsolete dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1070239 [13:20:07] (03CR) 10Hashar: [C:03+2] Remove obsolete dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1070239 (owner: 10Robert Vogel) [13:21:03] hashar: Tsk, don't merge patches with terrible commit titles. [13:21:04] It makes git log / git blame very messy. [13:21:26] (03PS2) 10Jforrester: Zuul: [BlueSpiceSocialWikiPage] Drop BlueSpiceEchoConnector dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1070239 (owner: 10Robert Vogel) [13:21:31] (03CR) 10Jforrester: [C:03+2] Zuul: [BlueSpiceSocialWikiPage] Drop BlueSpiceEchoConnector dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1070239 (owner: 10Robert Vogel) [13:21:40] James_F: argh too late [13:21:42] :D [13:21:50] but yeah good point [13:21:53] Not too late. :-) [13:21:56] that is also going to break that repo on REL branches [13:21:58] but well hmm [13:22:05] Yes, but that's their problem. [13:22:20] We defer to BlueSpice on how to manage that. [13:22:56] +1 [13:23:15] (03Merged) 10jenkins-bot: Zuul: [BlueSpiceSocialWikiPage] Drop BlueSpiceEchoConnector dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1070239 (owner: 10Robert Vogel) [13:24:50] I have deployed it [13:25:21] <3 [13:36:44] 10Continuous-Integration-Config, 07ci-test-error: Gate-and-submit for release branch 1.39 and 1.41 fails for Babel and GrowthExperiments because it tries to install CommunityConfiguration (which does not have a 1_39 or 1.41 branch) - https://phabricator.wikimedia.org/T373736#10112997 (10Jdforrester-WMF) It wil... [14:25:15] PROBLEM - SSH on doc2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [14:25:43] PROBLEM - SSH on gerrit1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [14:25:44] PROBLEM - SSH on gerrit2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [14:25:52] PROBLEM - SSH on doc1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [14:26:08] hmm [14:26:46] I guess monitoring is flappy [14:29:14] something being migrated [14:38:19] https://amarula.github.io/chatgpt-code-review-website/ [14:38:21] * hashar whistles [14:38:40] so then you get "developers" writing code with LLM which is then reviewed by LLM [14:38:50] amended by LLM and approved by LLM [14:40:12] I say we either: [14:40:12] A) approach The Singularity ™ https://en.wikipedia.org/wiki/Technological_singularity [14:40:12] B) lean toward realizing in real life the Idiocray prophecy https://en.wikipedia.org/wiki/Idiocracy [15:05:21] C) why not both D) the D answer [15:07:17] (03PS1) 10Effie Mouzeli: manifest.json,background.js: Add wikitech.wikimedia.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) [15:58:32] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T373640#10113718 (10dancy) [16:01:48] RECOVERY - SSH on gerrit1003 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [16:03:02] 10Gerrit (Gerrit 3.10): Upgrade to Gerrit 3.10.1 - https://phabricator.wikimedia.org/T373897 (10hashar) 03NEW [16:35:18] (03PS1) 10Subramanya Sastry: Work around jawikivoyage-specific visual diff noise [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1070299 [16:35:18] (03PS1) 10Subramanya Sastry: Fix bug in p-br-p stripping [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1070300 [16:38:25] hashar: eh yea, that was odd.. but I can confirm there is work on moving the alerting servers, denisse is on that [16:38:50] there was also an alert that generally the alerting server wasnt receiving data for a moment [16:38:52] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration: Enable CommunityConfiguration Example in CI - https://phabricator.wikimedia.org/T373114#10113968 (10thcipriani) Triage things: * [[h... [17:02:55] (03merge) 10jhuneidi: patchdemo: make and chown /var/www/.cache dir [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/65 (https://phabricator.wikimedia.org/T373721) [17:13:21] (03update) 10sandeeps: cli.py: add scap version to fatal error messages for better logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/407 (https://phabricator.wikimedia.org/T311080) [17:16:06] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: Switch Jenkins instances from Java 11 to Java 17 - https://phabricator.wikimedia.org/T359795#10114176 (10Dzahn) contint* machines now have both JDK 17 and JDK 11 installed: But 11 is still the default: ` [c... [17:16:19] (03update) 10sandeeps: cli.py: add scap version to fatal error messages for better logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/407 (https://phabricator.wikimedia.org/T311080) [17:20:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration: Enable CommunityConfiguration Example in CI - https://phabricator.wikimedia.org/T373114#10114191 (10Jdforrester-WMF) Copying my comme... [17:31:58] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration: Enable CommunityConfiguration Example in CI - https://phabricator.wikimedia.org/T373114#10114269 (10Michael) Thank you for the feedba... [17:49:19] (03CR) 10Ladsgroup: [C:03+1] "overall looks good to me. I think a message to wikitech-l letting people know that they might see weird results (domain not configured, et" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) (owner: 10Effie Mouzeli) [18:15:11] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T373640#10114567 (10dancy) [18:22:32] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T373640#10114591 (10dancy) [18:43:53] (03update) 10dduvall: make-release: Use the same credentials for Gerrit API and Git operations [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/111 (https://phabricator.wikimedia.org/T373441) [18:46:00] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T373640#10114670 (10Jdforrester-WMF) [18:47:56] James_F: Do you recommend rolling back due to https://phabricator.wikimedia.org/T373921 ? [18:48:49] oh , I see you making fixes. [18:55:58] dancy: Yeah, no need to roll back if I get rapid code review. ;-) [18:56:11] Hmm, actually, maybe we should? [18:56:38] I think I will to be on the safe side. [18:56:50] +1 [20:08:56] (03CR) 10Krinkle: [C:04-1] "I haven't tested it locally (see readme for how to), but I expect this not to work. It is already covered. the logic specifically skips th" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) (owner: 10Effie Mouzeli) [20:16:13] !log Updating development images on contint primary for https://phabricator.wikimedia.org/T373721 [20:16:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:18:45] 10Continuous-Integration-Infrastructure, 06Community-Tech, 10MediaWiki-extensions-CodeMirror, 07Documentation: CodeMirror docs generation failing post-merge - https://phabricator.wikimedia.org/T373937 (10MusikAnimal) 03NEW [20:20:54] 10Continuous-Integration-Config, 06Community-Tech, 10MediaWiki-extensions-CodeMirror, 07Documentation: CodeMirror docs generation failing post-merge - https://phabricator.wikimedia.org/T373937#10115088 (10MusikAnimal) [20:25:36] 10Fresh: Update Fresh's version of Node 18 from 18.20.2 to 18.20.4 - https://phabricator.wikimedia.org/T373938 (10MusikAnimal) 03NEW [20:31:16] (03update) 10dduvall: make-release: Use the same credentials for Gerrit API and Git operations [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/111 (https://phabricator.wikimedia.org/T373441) [20:36:58] (03PS1) 10MusikAnimal: fresh-node18: Upgrade to Node.js 18.20.4 from 18.20.2 [fresh] - 10https://gerrit.wikimedia.org/r/1070340 (https://phabricator.wikimedia.org/T373938) [20:40:28] (03CR) 10MusikAnimal: "FYI I was going by Ia4261ea15403dfb13004447b40e0a91ab7e0f880. I'm not sure if this change bumps the browser versions." [fresh] - 10https://gerrit.wikimedia.org/r/1070340 (https://phabricator.wikimedia.org/T373938) (owner: 10MusikAnimal) [20:44:45] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Figure out how to provision a Kubernetes cluster using Magnum and OpenTofu - https://phabricator.wikimedia.org/T372498#10115186 (10bd808) >>! In T372498#10079160, @bd808 wrote: > [] Figure out ingress for the Kubernetes cluster and add setup for that to the Op... [20:46:18] (03update) 10dduvall: make-release: Use the same credentials for Gerrit API and Git operations [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/111 (https://phabricator.wikimedia.org/T373441) [21:38:03] (03PS1) 10PixDeVl: Add PixDeVl to trusted users [integration/config] - 10https://gerrit.wikimedia.org/r/1070343 [21:40:10] (03CR) 10RhinosF1: [C:03+1] "trusted" [integration/config] - 10https://gerrit.wikimedia.org/r/1070343 (owner: 10PixDeVl) [21:42:24] (03update) 10dduvall: make-release: Use the same credentials for Gerrit API and Git operations [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/111 (https://phabricator.wikimedia.org/T373441) [21:43:34] (03update) 10dduvall: make-release: Use the same credentials for Gerrit API and Git operations [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/111 (https://phabricator.wikimedia.org/T373441) [21:50:49] (03approved) 10dancy: make-release: Use the same credentials for Gerrit API and Git operations [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/111 (https://phabricator.wikimedia.org/T373441) (owner: 10dduvall)