[00:00:08] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 46.67% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:24:09] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:52:58] 10Release-Engineering-Team (Watching / External), 10MediaWiki-API, 10Parsoid, 10monitoring, and 2 others: API action=parsoid-batch not available on Graphite - https://phabricator.wikimedia.org/T152776 (10tstarling) This should be fixed with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/447041 [01:41:59] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10Prtksxna) [05:22:59] !log Updated cxserver to d3c9d15 [05:23:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:57:41] 10Release-Engineering-Team (Watching / External), 10MediaWiki-API, 10Parsoid, 10monitoring, and 2 others: API action=parsoid-batch not available on Graphite - https://phabricator.wikimedia.org/T152776 (10hashar) 05Open>03Resolved a:03tstarling Thank you Tim [07:10:54] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<10.00%) [07:25:57] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [08:41:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10WMDE-leszek) For T199983 there is a fix in place that allows to move on with the train (in the light of that bug). The task stays open... [10:02:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T191060 (10zeljkofilipin) a:05hashar>03zeljkofilipin [10:03:29] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T191061 (10zeljkofilipin) a:05hashar>03mmodell [10:03:42] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T191061 (10zeljkofilipin) a:05mmodell>03None [10:04:14] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T191060 (10zeljkofilipin) [10:04:59] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10MediaWiki-Core-Tests, 10TCB-Team, 10User-zeljkofilipin: AdvancedSearch Selenium tests failing with `An element could not be located on the page using the given search parameters` - https://phabricator.wikimedia.org/T200211 (10zeljkofilipin) a:05z... [10:08:19] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10zeljkofilipin) [10:10:47] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10zeljkofilipin) Removing {T199983} from blockers because of T199983#4447103 and T191059#4447107. [12:02:38] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap sync` fails with `Error: You are missing some external dependencies.` - https://phabricator.wikimedia.org/T200257 (10zeljkofilipin) [12:02:40] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap sync` fails with `Error: You are missing some external dependencies.` - https://phabricator.wikimedia.org/T200257 (10zeljkofilipin) p:05Triage>03Unbreak! [12:04:07] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap sync` fails with `Error: You are missing some external dependencies.` - https://phabricator.wikimedia.org/T200257 (10zeljkofilipin) [12:04:32] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap sync` fails with `Error: You are missing some external dependencies.` - https://phabricator.wikimedia.org/T200257 (10zeljkofilipin) [12:04:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T191060 (10zeljkofilipin) [12:05:34] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:47:20] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap sync` fails with `Error: You are missing some external dependencies.` - https://phabricator.wikimedia.org/T200257 (10thcipriani) the `vendor` directory is missing. This is coming from MW not scap: https://github.com/wikimedia/mediawiki... [12:50:46] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T191060 (10thcipriani) [12:50:48] 10Release-Engineering-Team (Kanban), 10Scap, 10User-zeljkofilipin: `scap sync` fails with `Error: You are missing some external dependencies.` - https://phabricator.wikimedia.org/T200257 (10thcipriani) 05Open>03Resolved a:03thcipriani the branch checked out to /srv/mediawiki-staging/php-1.32.0-wmf.14 w... [12:53:07] Why it doesn't merge it? https://gerrit.wikimedia.org/r/c/design/landing-page/+/446890 :/ I can't find the submit button in the new UI [12:55:15] Amir1 submit won't show for you unless your a project owner or is a admin [12:55:22] since it inherits All-Projects [12:55:33] ohh, I forgot about it [12:55:35] right [12:57:15] (03PS1) 10Paladox: Add JenkinsBot group to submit on refs/* [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/447613 [13:03:17] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10thcipriani) [13:20:55] thcipriani nice @ the patch you made on gerrit-review! [13:47:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10Krinkle) [13:51:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T191061 (10mmodell) a:03mmodell [13:55:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Upstream, 10Wikimedia-log-errors (Shared Build Failure): Quibble CI jobs time out after 30min due to instance stalling at "npm install parse" step - https://phabricator.wikimedia.org/T198348 (10Krinkle) [14:23:39] (03CR) 10Krinkle: [C: 031] "I imagine the main difference will be that 'shallow-clone' in defaults.yaml uses 'wipe-workspace: true' instead of 'clean: after: true' (a" [integration/config] - 10https://gerrit.wikimedia.org/r/405722 (https://phabricator.wikimedia.org/T179963) (owner: 10Hashar) [14:24:51] (03CR) 10Krinkle: [C: 031] "Ah, the regular phplint job still uses shallow, but we created a separate one for mwgate and somehow skipped it. Maybe to make repeat buil" [integration/config] - 10https://gerrit.wikimedia.org/r/405722 (https://phabricator.wikimedia.org/T179963) (owner: 10Hashar) [14:29:18] (03CR) 10Krinkle: "Why? We normally add this at the project level, and even then only for refs/heads/. Is there a problem or task associated?" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/447613 (owner: 10Paladox) [14:30:17] (03CR) 10Paladox: "> Why? We normally add this at the project level, and even then only" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/447613 (owner: 10Paladox) [14:35:14] (03CR) 10Krinkle: [C: 04-1] "Indeed, mediawiki/ seems to have it on refs/*. I do not why." [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/447613 (owner: 10Paladox) [15:34:08] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10greg) [16:01:07] is the wmf.13 will be deployed everywhere today? [16:01:15] or it's completely halted? [16:03:21] Amir1: it's going to be unblocked shortly (thanks to anomie and addshore ) and then we'll move on to wmf.14 later. [16:06:29] greg-g: I merged the Anomie's patch, should I backported it? [16:07:10] Amir1: sure! cc zeljkof ^ [16:08:12] Amir1: please do! [16:10:05] one it [16:10:08] *on [17:03:32] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10zeljkofilipin) [17:25:41] 10Release-Engineering-Team (Watching / External), 10Phlogiston (Requests), 10User-greg: Adjust phlogiston configuration for Release Engineering - https://phabricator.wikimedia.org/T170359 (10JAufrecht) a:03greg These reports are up and stable and have been for most of the last year. http://phlogiston.wmf... [17:44:46] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T191059 (10zeljkofilipin) 05Open>03Resolved Deployed to all wikis. [17:45:37] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [18:08:12] (03PS1) 10Thcipriani: Add wikiversions-for-dblist tool [tools/release] - 10https://gerrit.wikimedia.org/r/447657 [18:15:29] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Jenkins, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 (10Dzahn) [18:15:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Jenkins, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 (10Dzahn) @Hashar Can we change Jenkins config to use the new host per Krinkle's question above? [18:41:10] (03CR) 10Greg Grossmeier: [C: 031] "Works as intended in my quick "make sure it runs and outputs the right numbers" test." [tools/release] - 10https://gerrit.wikimedia.org/r/447657 (owner: 10Thcipriani) [20:15:11] 10Release-Engineering-Team, 10Analytics-Kanban, 10DNS, 10Operations, and 5 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776 (10JAllemandou) [23:45:09] (03PS3) 10Paladox: Add Evad37 to the whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/447227