[00:11:46] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10brennen) I'm signing off for the day. Noting here that logs have been somewhat noisy for much of the day, likely due t... [00:22:39] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10DannyS712) >>! In T249962#6091834, @brennen wrote: > I'm signing off for the day. Noting here that logs have been some... [04:33:00] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): deployment-charts: Errors when deploying charts newly created from scaffold - https://phabricator.wikimedia.org/T251363 (10jeena) [05:12:14] 10Beta-Cluster-Infrastructure, 10JavaScript, 10User-DannyS712: Beta cluster: Recent changes returns `Internal error` - https://phabricator.wikimedia.org/T251364 (10DannyS712) [05:13:09] 10Beta-Cluster-Infrastructure, 10JavaScript, 10User-DannyS712: Beta cluster: Recent changes returns `Internal error` - https://phabricator.wikimedia.org/T251364 (10DannyS712) [05:14:46] 10Beta-Cluster-Infrastructure, 10JavaScript, 10User-DannyS712: Beta cluster: Recent changes returns `Internal error` - https://phabricator.wikimedia.org/T251364 (10DannyS712) This is **not** an issue at https://test.wikipedia.org/wiki/Special:RecentChanges currently running cbce38f; beta cluster is at 60961f7 [05:26:00] 10Beta-Cluster-Infrastructure, 10JavaScript, 10User-DannyS712: Beta cluster: Recent changes returns `Internal error` - https://phabricator.wikimedia.org/T251364 (10DannyS712) `default: mw.user.options.get( this.limitPreferenceName, displayConfig.limitDefault ),` - [here](https://gerrit.wikimedia.org/g/mediaw... [05:28:56] 10Beta-Cluster-Infrastructure, 10JavaScript, 10User-DannyS712: Beta cluster: changes list special pages return `Internal error` - https://phabricator.wikimedia.org/T251364 (10DannyS712) [05:50:59] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10DannyS712) [06:17:29] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [services/push-notifications] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593096 [06:17:31] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [services/push-notifications] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593096 (owner: 10QChris) [06:17:39] (03PS1) 10QChris: Import done. Revoke import grants [services/push-notifications] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593097 [06:17:41] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [services/push-notifications] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593097 (owner: 10QChris) [06:22:36] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Documentation, 10Kubernetes: Document how to migrate a service to kubernetes - https://phabricator.wikimedia.org/T248916 (10jeena) I have created documentation here: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial [06:22:44] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Documentation, 10Kubernetes: Document how to migrate a service to kubernetes - https://phabricator.wikimedia.org/T248916 (10jeena) 05Open→03Resolved [06:24:23] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [wikimedia/meet-accountmanager] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593099 [06:24:25] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [wikimedia/meet-accountmanager] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593099 (owner: 10QChris) [06:24:31] (03PS1) 10QChris: Import done. Revoke import grants [wikimedia/meet-accountmanager] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593100 [06:24:33] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [wikimedia/meet-accountmanager] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/593100 (owner: 10QChris) [07:31:26] 10Phabricator: @Phabricator_maintenance is sending email notifications - https://phabricator.wikimedia.org/T216867 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/593166 has been created. Please also see the comments on there. [07:51:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:53] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:12] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Epic, 10Tracking-Neverending: [EPIC] Provide pre-merge reports on patchsets (tracking) - https://phabricator.wikimedia.org/T101542 (10hashar) [07:54:14] 10Continuous-Integration-Infrastructure, 10Epic: Provide (pre-merge) performance reports on patchsets - https://phabricator.wikimedia.org/T101543 (10hashar) 05Stalled→03Resolved a:03Krinkle It seems to me that is now fulfilled by #fresnel (written by @Krinkle) which lets one compare web performance repor... [07:56:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:09] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing: Use strict mode when running hhvm tests on jenkins - https://phabricator.wikimedia.org/T132270 (10hashar) 05Open→03Declined We no more use #HHVM There might have been some parameter to make it stricter. Anyway this task is now obsolete. [08:02:06] 10Continuous-Integration-Infrastructure: Some docker slave still have old containers using old images - https://phabricator.wikimedia.org/T176623 (10hashar) 05Open→03Resolved a:03dduvall That one got addressed by @dduvall which refactored the jobs to use `docker stop` to kill those containers instead of re... [08:02:59] 10Continuous-Integration-Infrastructure, 10Zuul: Zuul's web-ui expands multiple pipelines simultaniously - https://phabricator.wikimedia.org/T200466 (10hashar) [08:10:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer: Auto-fix errors in pushed changesets where possible - https://phabricator.wikimedia.org/T200790 (10hashar) 05Open→03Declined Use: `composer lint` `composer fix` And pass them the list of files you are touching. [08:29:22] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10Dzahn) Hi Lars, is the last part still a TODO? [08:33:23] (03PS1) 10JMeybohm: Debian glue for operations/debs/helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/593190 [08:35:01] Can someone check what's up with beta? [08:36:13] 10Phabricator, 10DBA, 10Operations: replace phabricator db passwords with longer passwords - https://phabricator.wikimedia.org/T250361 (10Dzahn) [08:36:52] deploymentwikimedia is ERR_CONNECTION_CLOSED [08:37:25] (03CR) 10JMeybohm: Debian glue for operations/debs/helm3 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/593190 (owner: 10JMeybohm) [08:46:04] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52135 bytes in 1.401 second response time [08:46:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92482 bytes in 0.934 second response time [08:47:46] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 92796 bytes in 1.065 second response time [08:57:08] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10Dzahn) 05Open→03Resolved Disk space alert is OK since almost a month, i gzipped the existing logs and beyond that i don't think it's worth the... [08:57:11] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) [09:00:02] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) a:05Dzahn→03QChris @QChris fyi, this is the dedicated test machine for the gerrit upgrade, you can feel free to use it. I confirmed your shell user exists. also see T243808#60257... [09:02:10] 10Release-Engineering-Team-TODO, 10Operations: Should 'doc' machines (i.e. doc1001) have contint-roots as a group? - https://phabricator.wikimedia.org/T245691 (10Dzahn) 05Stalled→03Declined Alright, based on the last comment i will call it declined then. Reopen if the need arises, of course. [09:08:31] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing: Add CI entry point to run MinusX on mediawiki/core - https://phabricator.wikimedia.org/T188022 (10hashar) [09:09:00] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing: Add CI entry point to run MinusX on mediawiki/core - https://phabricator.wikimedia.org/T188022 (10hashar) [09:09:02] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Run minus-x on mediawiki/core for all files - https://phabricator.wikimedia.org/T212746 (10hashar) [09:09:40] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing: Add CI entry point to run MinusX on mediawiki/core - https://phabricator.wikimedia.org/T188022 (10hashar) [09:10:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Puppet fails on Beta Cluster because "did not find a value for the name 'profile::envoy::ensure'" - https://phabricator.wikimedia.org/T247147 (10Dzahn) >>! In T247147#5962431, @cscott wrote: > I'm still fuzzy on how Horiz... [09:12:12] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing: Add CI entry point to run MinusX on mediawiki/core - https://phabricator.wikimedia.org/T188022 (10hashar) `minus-x` had been added to `composer test` and CI passes it the list of php files modified. That thus prevented `minus-x` to run against other k... [09:14:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Assess whether we should still disable seccomp in Docker for CI - https://phabricator.wikimedia.org/T249729 (10hashar) 05Open→03Resolved a:03hashar Per my previo... [09:17:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Operations: Assess whether we should still disable seccomp in Docker for CI - https://phabricator.wikimedia.org/T249729 (10MoritzMuehlenhoff) 05Resolved→03Open I'm reopening t... [09:24:03] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Developer Productivity, 10Epic: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10Dzahn) + @QChris [09:29:34] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Wikibugs: jerkins-bot should not post on IRC for Gerrit changes marked 'WIP' - https://phabricator.wikimedia.org/T239928 (10hashar) The notifications are emitted by #wikibugs https://gerrit.wikimedia.org/r/plugins/gitiles... [09:35:33] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) If it's just log noise, we can wait for next week's train I think. [09:36:37] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) I've woken up and logstash for the past 12 hours seems reasonably calm. [10:10:43] 10Release-Engineering-Team, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team), 10Performance Issue, 10Wikimedia-database-error: WikiPage::updateCategoryCounts causing replication lag due to long-running writes on commonswiki - https://phabricator.wikimedia.org/T240405 (10Aklapper) @CCicale... [10:15:00] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10LarsWirzenius) @Dzahn yeah, I've not gotten signed keys yet, or if I had, I'm too scatter brained to remember it. [10:17:15] 10Beta-Cluster-Infrastructure, 10JavaScript, 10User-DannyS712: Beta cluster: changes list special pages return Internal error: TypeError: Cannot read property 'limitDefault' of null - https://phabricator.wikimedia.org/T251364 (10Aklapper) [10:26:04] 10Continuous-Integration-Config: some jjb inline bash snippets might miss set -eu - https://phabricator.wikimedia.org/T106384 (10hashar) 05Open→03Declined That is less of an issue nowadays. [10:36:39] (03CR) 10Hashar: [C: 03+2] "Note we already have repository for helm: operations/debs/helm But maybe it is too complicated to have to maintain multiple Debian branch" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/593190 (owner: 10JMeybohm) [10:37:01] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10Nikerabbit) [10:37:35] (03Merged) 10jenkins-bot: Debian glue for operations/debs/helm3 [integration/config] - 10https://gerrit.wikimedia.org/r/593190 (owner: 10JMeybohm) [11:23:34] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-Recent-changes, 10JavaScript, 10User-DannyS712: Beta cluster: changes list special pages return Internal error: TypeError: Cannot read property 'limitDefault' of null - https://phabricator.wikimedia.org/T251364 (10Aklapper) [11:25:53] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10Aklapper) [11:29:26] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10DannyS712) [11:34:23] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10Dzahn) fyi, recently keyservers started stripping signatures. So if you use --recv-keys and don't see any with --list-sigs that is why. see T242309#5872124 -> /T2423... [11:42:53] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10DannyS712) @LarsWirzenius just in case you miss it (better safe than sorry): {T251404} may not be showing up on logstas... [11:55:51] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-Recent-changes, 10JavaScript, 10User-DannyS712: Beta cluster: changes list special pages return Internal error: TypeError: Cannot read property 'limitDefault' of null - https://phabricator.wikimedia.org/T251364 (10DannyS712) [11:56:14] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-Recent-changes, 10JavaScript, 10User-DannyS712: Beta cluster: changes list special pages return Internal error: TypeError: Cannot read property 'limitDefault' of null - https://phabricator.wikimedia.org/T251364 (10DannyS712) Closing as duplicate -... [11:59:05] (03PS6) 10DannyS712: UnusedUseStatementSniff: Recognize uses in `@phan-var` comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591600 (https://phabricator.wikimedia.org/T250765) [12:22:53] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) @DannyS712 thanks; I've commented on that task, but to summarize: I don't think it's a blocker for group... [12:37:22] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Joe) >>! In T99740#6089626, @Ladsgroup wrote: >>>! In T99740#6089198, @Joe wro... [12:48:59] (03CR) 10JMeybohm: "> Patch Set 1: Code-Review+2" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/593190 (owner: 10JMeybohm) [12:49:16] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) [13:06:14] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [13:06:29] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10LarsWirzenius) group1 is at 1.35.30-wmf.30 now [13:09:49] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%) [13:11:18] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [13:19:52] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [13:32:04] 10Release-Engineering-Team, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team), 10Performance Issue, 10Wikimedia-database-error: WikiPage::updateCategoryCounts causing replication lag due to long-running writes on commonswiki - https://phabricator.wikimedia.org/T240405 (10CCicalese_WMF) Per... [13:41:26] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: Unable to clone Mediawiki core repo due to timeout - https://phabricator.wikimedia.org/T249410 (10zeljkofilipin) Was this resolved, or is it still a problem? [13:43:34] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: Unable to clone Mediawiki core repo due to timeout - https://phabricator.wikimedia.org/T249410 (10zeljkofilipin) a:03AlQaholic007 [13:45:02] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: Unable to clone Mediawiki core repo due to timeout - https://phabricator.wikimedia.org/T249410 (10AlQaholic007) Hi @zeljkofilipin this is still unresolved. I am not too sure if it's a network issue but I was able to work around tempor... [13:59:21] (03CR) 10Hashar: [C: 03+2] "Yes, it is probably easier than having to deal with multiple branches that end up weirdly named ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/593190 (owner: 10JMeybohm) [14:00:45] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10MediaWiki-extensions-CodeReview, 10Wikimedia-Site-requests, 10Technical-Debt: Undeploy CodeReview - https://phabricator.wikimedia.org/T116948 (10CCicalese_WMF) [14:03:56] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10DannyS712) [14:20:48] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<10.00%) [14:24:19] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: Unable to clone Mediawiki core repo due to timeout - https://phabricator.wikimedia.org/T249410 (10hashar) Can you try again with some debug logs turned on? GIT_TRACE=1 git clone ssh://alqaholic007@gerrit.wikimedia.org:29418/sandbo... [14:28:25] 10Release-Engineering-Team-TODO, 10Packaging, 10Upstream: gbp buildpackage with GIT_PBUILDER_AUTOCONF=no causes DIST to be ignored - https://phabricator.wikimedia.org/T233020 (10JMeybohm) [14:30:47] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [14:31:26] (03CR) 10Joal: Port analytics-update-jars to Docker (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/589589 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [14:37:00] (03PS1) 10Hashar: Remove .arcconfig add .gitreview [tools/scap-vagrant] - 10https://gerrit.wikimedia.org/r/593243 (https://phabricator.wikimedia.org/T216483) [14:37:53] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, and 2 others: Move scap-vagrant to Gerrit - https://phabricator.wikimedia.org/T216483 (10hashar) I have created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/tools/scap-vagrant which inh... [14:37:56] 10Release-Engineering-Team-TODO, 10Packaging, 10Upstream: gbp buildpackage with GIT_PBUILDER_AUTOCONF=no causes DIST to be ignored - https://phabricator.wikimedia.org/T233020 (10JMeybohm) As @Ottomata and @akosiaris writing in T250803 this hits on deneb again and the approach from https://bugs.debian.org/cgi... [14:43:57] (03PS7) 10Hashar: Port analytics-update-jars to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/589589 (https://phabricator.wikimedia.org/T210271) [14:50:45] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, and 2 others: Move scap-vagrant to Gerrit - https://phabricator.wikimedia.org/T216483 (10hashar) 05Open→03Resolved a:03hashar I have managed to enable pulling from Gerrit :) https://phabricat... [14:50:49] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Stop using Differential for code review - https://phabricator.wikimedia.org/T191182 (10hashar) [14:56:11] (03PS8) 10Hashar: Port analytics-update-jars to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/589589 (https://phabricator.wikimedia.org/T210271) [14:56:13] (03PS1) 10Hashar: docker: jar-updater set WORKDIR to /src [integration/config] - 10https://gerrit.wikimedia.org/r/593248 [14:57:18] (03CR) 10Hashar: [C: 03+2] docker: jar-updater set WORKDIR to /src [integration/config] - 10https://gerrit.wikimedia.org/r/593248 (owner: 10Hashar) [14:58:15] (03Merged) 10jenkins-bot: docker: jar-updater set WORKDIR to /src [integration/config] - 10https://gerrit.wikimedia.org/r/593248 (owner: 10Hashar) [14:58:34] hashar: not urgent: can i have a review on https://gerrit.wikimedia.org/r/c/operations/puppet/+/591338 https://gerrit.wikimedia.org/r/c/operations/dns/+/591340 [14:59:56] mutante: ah yeah [15:02:04] mutante: +1ed both [15:03:26] thanks hashar [15:08:07] 10Release-Engineering-Team-TODO, 10Packaging, 10Upstream: gbp buildpackage with GIT_PBUILDER_AUTOCONF=no causes DIST to be ignored - https://phabricator.wikimedia.org/T233020 (10JMeybohm) Using `GIT_PBUILDER_AUTOCONF=no gbp buildpackage -sa -us -uc --git-pbuilder --git-dist=stretch` instead of `GIT_PBUILDER... [15:34:20] (03PS9) 10Hashar: Port analytics-update-jars to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/589589 (https://phabricator.wikimedia.org/T210271) [15:35:45] (03CR) 10Hashar: [C: 03+2] "Paired with joal :)" [integration/config] - 10https://gerrit.wikimedia.org/r/589589 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [15:36:20] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) [15:36:46] (03Merged) 10jenkins-bot: Port analytics-update-jars to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/589589 (https://phabricator.wikimedia.org/T210271) (owner: 10Hashar) [15:37:48] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10hashar) [15:37:53] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar... [15:39:37] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10hashar) The last tied job is https://integration.wikimed... [15:43:00] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10DannyS712) >>! In T249962#6093218, @LarsWirzenius wrote: > @DannyS712 thanks; I've commented on that task, but to summa... [15:55:17] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10JAllem... [15:55:24] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Analytics, and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10JAllem... [15:57:04] (03PS1) 10Majavah: Run tox-docker in the wikimedia/meet-accountmanager repository [integration/config] - 10https://gerrit.wikimedia.org/r/593266 (https://phabricator.wikimedia.org/T251425) [15:57:20] hashar: Hurrah, https://integration.wikimedia.org/ci/label/DebianJessie/ is empty! [15:59:03] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10Jdforrester-WMF) 05Open→03Resolved >>! In T239981#60... [15:59:05] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [15:59:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10Jdforrester-WMF) [16:01:54] (03CR) 10Ladsgroup: [C: 03+1] Run tox-docker in the wikimedia/meet-accountmanager repository [integration/config] - 10https://gerrit.wikimedia.org/r/593266 (https://phabricator.wikimedia.org/T251425) (owner: 10Majavah) [16:03:01] !log Removed integration-slave-jessie-1002 and integration-slave-jessie-1004 from Jenkins for T236576 [16:03:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:03:03] T236576: "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 [16:03:51] !log Shut down integration-slave-jessie-1002 and integration-slave-jessie-1004 in Horizon for T236576 [16:03:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:06:28] maryum: hashar should know about configuring the git auth for your jenkins jobs. Hopefully. :-) [16:07:03] I'll add a comment in the phab ticket as well [16:08:16] PROBLEM - Host integration-slave-jessie-1002 is DOWN: CRITICAL - Host Unreachable (172.16.1.99) [16:09:05] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Wikidata, 10Wikidata-Campsite, 10Wikidata-Query-Service, and 2 others: Migrate wikidata-query-rdf-release-silent release job to Docker - https://phabricator.wikimedia.org/T247123 (10Mstyles) @hashar the jenkins job failed due to no git auth to p... [16:09:12] James_F: going to reopen that ticket if that's okay [16:09:16] PROBLEM - Host integration-slave-jessie-1004 is DOWN: CRITICAL - Host Unreachable (172.16.2.228) [16:09:26] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10Mstyles) [16:09:32] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Wikidata, 10Wikidata-Campsite, 10Wikidata-Query-Service, and 2 others: Migrate wikidata-query-rdf-release-silent release job to Docker - https://phabricator.wikimedia.org/T247123 (10Mstyles) 05Resolved→03Open [16:09:37] maryum: Sure. [16:13:00] (03CR) 10Jforrester: [C: 03+2] Run tox-docker in the wikimedia/meet-accountmanager repository [integration/config] - 10https://gerrit.wikimedia.org/r/593266 (https://phabricator.wikimedia.org/T251425) (owner: 10Majavah) [16:13:00] PROBLEM - Host integration-trigger-01 is DOWN: CRITICAL - Host Unreachable (172.16.6.6) [16:14:02] (03Merged) 10jenkins-bot: Run tox-docker in the wikimedia/meet-accountmanager repository [integration/config] - 10https://gerrit.wikimedia.org/r/593266 (https://phabricator.wikimedia.org/T251425) (owner: 10Majavah) [16:14:38] !log Zuul: CI for wikimedia/meet-accountmanager T251425 [16:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:14:40] T251425: Run tox+flake8 tests on meet-accountmanager - https://phabricator.wikimedia.org/T251425 [16:19:17] 10Gerrit, 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: Unable to clone Mediawiki core repo due to timeout - https://phabricator.wikimedia.org/T249410 (10Aklapper) I boldly created https://www.mediawiki.org/wiki/Gerrit/Troubleshooting#git_clone - feel free to edit/improve. [16:21:14] maryum: I will catch up later tonight on the ticket. I am out for a walk with the kids [16:21:16] :) [16:21:21] thanks!! [16:21:57] * hashar vanishes [16:27:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:47] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92492 bytes in 1.088 second response time [16:50:59] (03PS1) 10Mstyles: jjb: add wdqs site publish job back [integration/config] - 10https://gerrit.wikimedia.org/r/593274 [17:07:03] James_F: I realized that I shouldn't have removed the site publish job. so then, https://gerrit.wikimedia.org/r/c/integration/config/+/593274 [17:08:34] * James_F nods. [17:47:28] 10MediaWiki-Codesniffer, 10User-DannyS712: Add a sniff for non-global variables named like globals - https://phabricator.wikimedia.org/T251443 (10DannyS712) [17:49:56] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): deployment-charts: Errors when deploying charts newly created from scaffold - https://phabricator.wikimedia.org/T251363 (10jeena) 05Open→03Resolved [17:55:35] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: Add extension to release branch cut - https://phabricator.wikimedia.org/T251442 (10Jdforrester-WMF) [18:01:26] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: Add extension to release branch cut - https://phabricator.wikimedia.org/T251442 (10Jdforrester-WMF) [18:03:01] (03PS38) 10DannyS712: Testing [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591409 [18:05:06] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10brennen) [18:06:35] (03PS39) 10DannyS712: Testing [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591409 [18:09:10] (03PS40) 10DannyS712: Testing [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591409 [18:16:30] (03CR) 10jerkins-bot: [V: 04-1] Testing [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/591409 (owner: 10DannyS712) [18:58:56] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:49] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92492 bytes in 1.041 second response time [19:56:31] maryum: back around ;) [19:56:39] the fix is reasonsably easy: we gotta push over https [19:56:41] not ssh [19:57:55] I don't even see where that happens in the yaml. it's using the https url for the repo [19:58:05] wait that's for zuul, never mind [20:00:54] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Wikidata, 10Wikidata-Campsite, 10Wikidata-Query-Service, and 2 others: Migrate wikidata-query-rdf-release-silent release job to Docker - https://phabricator.wikimedia.org/T247123 (10hashar) The job fails with: `counterexample [ERROR] Failed to e... [20:02:01] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Wikidata, 10Wikidata-Campsite, 10Wikidata-Query-Service, and 2 others: Migrate wikidata-query-rdf-release-silent release job to Docker - https://phabricator.wikimedia.org/T247123 (10hashar) >>! In T247123#6095073, @hashar wrote:` > **Next we wil... [20:02:07] maryum: I have crafted a patch for the pom.xml :) [20:02:17] https://gerrit.wikimedia.org/r/#/c/wikidata/query/rdf/+/593298 [20:02:25] I have done the same hack for analytics/refinery/source [20:02:35] I am going to update the job to pass -DdeveloperConnection="scm:git:$ZUUL_URL/$ZUUL_PROJECT" [20:02:48] which will then be used in the pom.xml to change the url [20:02:58] (03PS1) 10Umherirrender: Code cleanup: Remove unused variable and fields [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/593299 [20:04:57] (03PS1) 10Hashar: Override scm url for wikidata/query/rdf [integration/config] - 10https://gerrit.wikimedia.org/r/593300 (https://phabricator.wikimedia.org/T247123) [20:14:18] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [20:14:59] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [20:15:32] (03PS1) 10Umherirrender: Update squizlabs/php_codesniffer to 3.5.5 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/593301 [20:15:57] (03PS2) 10Hashar: Override scm url for wikidata/query/rdf [integration/config] - 10https://gerrit.wikimedia.org/r/593300 (https://phabricator.wikimedia.org/T247123) [20:15:59] (03PS1) 10Hashar: Remove wikidata/query/rdf release job from postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/593302 (https://phabricator.wikimedia.org/T247123) [20:16:01] (03PS1) 10Hashar: Restore wikidata/query/rdf jobs [integration/config] - 10https://gerrit.wikimedia.org/r/593303 [20:17:24] (03CR) 10Hashar: "That change had a few issues, I have made follow up changes:" [integration/config] - 10https://gerrit.wikimedia.org/r/591522 (https://phabricator.wikimedia.org/T247123) (owner: 10Mstyles) [20:18:16] (03PS2) 10Hashar: Restore wikidata/query/rdf jobs [integration/config] - 10https://gerrit.wikimedia.org/r/593303 (https://phabricator.wikimedia.org/T247123) [20:19:59] (03CR) 10Hashar: [C: 03+2] Remove wikidata/query/rdf release job from postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/593302 (https://phabricator.wikimedia.org/T247123) (owner: 10Hashar) [20:20:56] (03Merged) 10jenkins-bot: Remove wikidata/query/rdf release job from postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/593302 (https://phabricator.wikimedia.org/T247123) (owner: 10Hashar) [20:21:10] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10local-charts: local-charts: Repair ability to enable xdebug on mw/core - https://phabricator.wikimedia.org/T246921 (10jeena) 05Open→03Resolved [20:24:01] (03CR) 10Hashar: [C: 03+2] "I have recreated wikidata-query-rdf-maven-java8-docker-site-publish" [integration/config] - 10https://gerrit.wikimedia.org/r/593303 (https://phabricator.wikimedia.org/T247123) (owner: 10Hashar) [20:24:17] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [20:24:38] (03CR) 10Hashar: [C: 03+2] "And updated wikidata-query-rdf-maven-java8-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/593303 (https://phabricator.wikimedia.org/T247123) (owner: 10Hashar) [20:24:49] (03PS1) 10Jeena Huneidi: values.example.yaml: Rename restbase to restrouter [releng/local-charts] - 10https://gerrit.wikimedia.org/r/593304 [20:25:01] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [20:25:01] (03Merged) 10jenkins-bot: Restore wikidata/query/rdf jobs [integration/config] - 10https://gerrit.wikimedia.org/r/593303 (https://phabricator.wikimedia.org/T247123) (owner: 10Hashar) [20:25:07] (03CR) 10Hashar: [C: 03+1] Override scm url for wikidata/query/rdf [integration/config] - 10https://gerrit.wikimedia.org/r/593300 (https://phabricator.wikimedia.org/T247123) (owner: 10Hashar) [20:27:02] hashar: thanks for the patch! [20:32:58] (03Abandoned) 10Jforrester: jjb: add wdqs site publish job back [integration/config] - 10https://gerrit.wikimedia.org/r/593274 (owner: 10Mstyles) [20:42:17] Jdlrobson: oops [20:42:24] yeah duped things a bit sorry [20:42:30] Jdlrobson: wrong ping sorry! [20:43:26] maryum: we will just need the pom.xml to be slightly adjusted which is https://gerrit.wikimedia.org/r/#/c/wikidata/query/rdf/+/593298/ [20:44:25] maryum: also Joseph (joal) pointed me the documentation they use for their release: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Deploy/Refinery-source [20:44:37] that might be a good read or worth copy pasting for wikidata/query/rdf [20:44:49] hashar: thanks, I'll wait until gehel approves the pom change [20:44:55] and thanks for the docs! [20:45:39] sure ;) [21:01:17] (03PS1) 10DC Slagel: Blubber: move expansion before verification [blubber] - 10https://gerrit.wikimedia.org/r/593315 (https://phabricator.wikimedia.org/T248927) [21:10:54] (03CR) 10DC Slagel: "This is an initial attempt and solving T248927: Blubber policy should be verified after expansion." [blubber] - 10https://gerrit.wikimedia.org/r/593315 (https://phabricator.wikimedia.org/T248927) (owner: 10DC Slagel) [21:12:23] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10cscott) Cross-posting from {T251409} -- that will cause an `Errore irreversibile di tipo "TypeError"` on pages which co... [21:18:54] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T249962 (10Jdforrester-WMF) [21:25:40] (03CR) 10Gehel: [C: 04-1] Override scm url for wikidata/query/rdf (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/593300 (https://phabricator.wikimedia.org/T247123) (owner: 10Hashar) [23:22:08] (03PS1) 10Jforrester: Stop branching the Sentry extension for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/593346 (https://phabricator.wikimedia.org/T91649) [23:33:59] (03PS1) 10Jforrester: Stop branching the CodeReview extension for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/593353 (https://phabricator.wikimedia.org/T116948) [23:46:49] (03CR) 10Krinkle: [C: 03+1] "LGTM!" [integration/reporting] - 10https://gerrit.wikimedia.org/r/588440 (owner: 10Dduvall) [23:54:07] (03PS1) 10Jforrester: layout: [mediawiki/extensions/Sentry] Move out of deployed section [integration/config] - 10https://gerrit.wikimedia.org/r/593355 (https://phabricator.wikimedia.org/T91649) [23:56:13] (03CR) 10Jforrester: [C: 03+2] Stop branching the Sentry extension for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/593346 (https://phabricator.wikimedia.org/T91649) (owner: 10Jforrester) [23:57:11] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/extensions/Sentry] Move out of deployed section [integration/config] - 10https://gerrit.wikimedia.org/r/593355 (https://phabricator.wikimedia.org/T91649) (owner: 10Jforrester) [23:58:05] (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/Sentry] Move out of deployed section [integration/config] - 10https://gerrit.wikimedia.org/r/593355 (https://phabricator.wikimedia.org/T91649) (owner: 10Jforrester) [23:59:04] !log Zuul: [mediawiki/extensions/Sentry] Move out of deployed section T91649 [23:59:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:59:07] T91649: Deploy Sentry (JavaScript error logging) to production, configured to log only a limited subset of users/pages - https://phabricator.wikimedia.org/T91649