[00:01:57] 10Beta-Cluster-Infrastructure: Generate beta captchas - https://phabricator.wikimedia.org/T204611 (10Krenair) Okay so it's fine with 952. It'll give you one swift error for every one over that. [01:25:04] Project beta-scap-eqiad build #222569: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222569/ [01:39:14] Yippee, build fixed! [01:39:15] Project beta-scap-eqiad build #222570: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222570/ [02:21:29] 10Phabricator, 10Cleanup: Form 33 amendment request - https://phabricator.wikimedia.org/T204487 (10Dinoguy1000) 05Resolved>03Open The form itself still has the old content; only the notice above the form was changed. [02:39:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [02:44:18] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [02:55:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [03:22:18] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.30-release, 10Security: MediaWiki 1.30.0 tarball is missing vendor/.htaccess - https://phabricator.wikimedia.org/T191597 (10Legoktm) 05Open>03Resolved a:03Reedy I forgot about this task, but it was fixed in 1.30.1. [03:22:28] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.30-release, 10Security: MediaWiki 1.30.0 tarball is missing vendor/.htaccess - https://phabricator.wikimedia.org/T191597 (10Legoktm) [04:19:43] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.31-release, 10Patch-For-Review: Upgrade patches for tarball releases don't apply cleanly to tarball installation - https://phabricator.wikimedia.org/T73379 (10Legoktm) 05Open>03Resolved a:03Legoktm [04:19:51] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools, 10Patch-For-Review: Make make-release not need to clone MW for every branch - https://phabricator.wikimedia.org/T180522 (10Legoktm) 05Open>03Resolved [04:19:55] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: make-release vendor diffing broken for REL1_27/REL1_28 - https://phabricator.wikimedia.org/T180532 (10Legoktm) 05Open>03Resolved [04:20:05] 10MediaWiki-Releasing: Automate Testing of MediaWiki Tarball releases - https://phabricator.wikimedia.org/T974 (10Legoktm) [05:29:00] (03PS2) 10Legoktm: Make MinervaNeue run seccheck non-voting. [integration/config] - 10https://gerrit.wikimedia.org/r/461510 (https://phabricator.wikimedia.org/T204094) (owner: 10Brian Wolff) [05:29:05] (03CR) 10Legoktm: [C: 032] Make MinervaNeue run seccheck non-voting. [integration/config] - 10https://gerrit.wikimedia.org/r/461510 (https://phabricator.wikimedia.org/T204094) (owner: 10Brian Wolff) [05:30:44] (03Merged) 10jenkins-bot: Make MinervaNeue run seccheck non-voting. [integration/config] - 10https://gerrit.wikimedia.org/r/461510 (https://phabricator.wikimedia.org/T204094) (owner: 10Brian Wolff) [05:32:29] !log deployed https://gerrit.wikimedia.org/r/461510 [05:32:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:49:47] (03PS1) 10Legoktm: Use more robust check to identify tags [tools/release] - 10https://gerrit.wikimedia.org/r/461859 [05:49:49] (03PS1) 10Legoktm: Require that tags are GPG signed [tools/release] - 10https://gerrit.wikimedia.org/r/461860 (https://phabricator.wikimedia.org/T204996) [05:49:51] (03PS1) 10Legoktm: Create a tar with the tarballs in it for easy upload [tools/release] - 10https://gerrit.wikimedia.org/r/461861 (https://phabricator.wikimedia.org/T205004) [06:16:30] 10Phabricator, 10Cleanup: Form 33 amendment request - https://phabricator.wikimedia.org/T204487 (10Aklapper) 05Open>03Resolved I should have used https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/defaults/33/ instead of https://phabricator.wikimedia.org/transactions/editengine/maniph... [06:28:11] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:28:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [06:29:50] 10Project-Admins, 10Performance-Team: Create project component WebPageReplay - https://phabricator.wikimedia.org/T205058 (10Peter) [06:33:16] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [06:35:27] 10Phabricator, 10User-MModell: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) @mgchecker: any custom fields which belong to one type would be lost when converting to another type. Converting a release task to any other type would completely break the release schedule and "ta... [06:39:18] 10Phabricator, 10Performance: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10mmodell) >>! In T108586#4602981, @Quiddity wrote: > Is it the number of //Rules// that is the problem, or more the number of //Conditions//? I.e. would it help if we aske... [06:44:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [06:45:24] 10Phabricator, 10Performance: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10mmodell) I was curious how much antivandalism was slowing things down so I disabled the rule and then checked transcripts on transactions that happened while the rule was... [06:49:17] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [06:50:09] PROBLEM - Puppet errors on deployment-certcentral-testdns is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [06:55:18] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [07:08:13] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:11] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10MW-1.32-release-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)), and 3 others: Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137 (10Addshore) [07:52:18] 10Project-Admins, 10Performance-Team: Create project component WebPageReplay - https://phabricator.wikimedia.org/T205058 (10Aklapper) 05Open>03Resolved a:03Aklapper Requested public project #WebPageReplay has been created: https://phabricator.wikimedia.org/project/view/3619/ Please encourage interested... [07:54:04] 10Project-Admins, 10Performance-Team: Create project component WebPageReplay - https://phabricator.wikimedia.org/T205058 (10Peter) Thank you @Aklapper ! [08:11:27] 10Release-Engineering-Team, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata: Undeploy WikibaseQuality extension from the WMF - https://phabricator.wikimedia.org/T205064 (10Addshore) [08:11:41] 10Release-Engineering-Team, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata: Undeploy WikibaseQuality extension from the WMF - https://phabricator.wikimedia.org/T205064 (10Addshore) 05Open>03stalled [08:12:08] 10Release-Engineering-Team, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata: Undeploy WikibaseQuality extension from the WMF - https://phabricator.wikimedia.org/T205064 (10Addshore) [08:12:52] 10Release-Engineering-Team, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata: Undeploy WikibaseQuality extension from the WMF - https://phabricator.wikimedia.org/T205064 (10Addshore) p:05Triage>03Normal [10:00:04] 10Phabricator, 10User-MModell: Add support for task types - https://phabricator.wikimedia.org/T93499 (10MGChecker) >>! In T93499#4604839, @mmodell wrote: > @mgchecker: any custom fields which belong to one type would be lost when converting to another type. Converting a release task to any other type would com... [10:01:50] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [10:24:30] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10hashar) A quick and dirty way to get the busiest repos per number of commits is to use git log on mediawiki/extensions: {P757... [10:25:50] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10hashar) Another note from a discussion with Zeljko: we have a third party gathering git/gerrit metrics. They store it in Elast... [10:38:45] (03CR) 10Addshore: [C: 031] Add back Selenium tests for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/457675 (https://phabricator.wikimedia.org/T189762) (owner: 10Hashar) [10:49:40] hashar: hey, if you think it's UBN, I can ask for an unscheduled deploy: https://phabricator.wikimedia.org/T204961 [10:51:49] Amir1: na I don't think that is ubn [10:52:12] Amir1: seems that happens whenever we switch version, I guess ORES can't catch up with the sudden wikis changes [10:52:31] so surely it should be fixed, but I don't think it is super urgent :] [10:52:56] I merely listed it as a blocker as a remember that it got noticed during the train [10:53:57] Oh okay, I will deploy the whitelisting part in Monday. [10:54:00] I commented on it [10:54:04] That would fix most of issues [10:54:06] yeah we can swat it on monday [10:54:09] Thanks [10:54:16] then on tuesday when doing the train there should be no more timeout :] [10:54:20] (hopefully) [10:56:33] (03PS1) 10Hashar: operations/debs/wmf-pt-kill add debian glue (non voting) [integration/config] - 10https://gerrit.wikimedia.org/r/461934 [10:57:07] (03CR) 10Hashar: [C: 032] operations/debs/wmf-pt-kill add debian glue (non voting) [integration/config] - 10https://gerrit.wikimedia.org/r/461934 (owner: 10Hashar) [10:59:34] (03Merged) 10jenkins-bot: operations/debs/wmf-pt-kill add debian glue (non voting) [integration/config] - 10https://gerrit.wikimedia.org/r/461934 (owner: 10Hashar) [11:05:32] (03CR) 10Zfilipin: [C: 031] "I think @hashar needs to make a new quibble release." [integration/quibble] - 10https://gerrit.wikimedia.org/r/461195 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [12:08:33] (03PS1) 10Hashar: Revert "operations/debs/wmf-pt-kill add debian glue (non voting)" [integration/config] - 10https://gerrit.wikimedia.org/r/461944 [12:08:45] (03CR) 10Hashar: [C: 032] Revert "operations/debs/wmf-pt-kill add debian glue (non voting)" [integration/config] - 10https://gerrit.wikimedia.org/r/461944 (owner: 10Hashar) [12:10:14] (03Merged) 10jenkins-bot: Revert "operations/debs/wmf-pt-kill add debian glue (non voting)" [integration/config] - 10https://gerrit.wikimedia.org/r/461944 (owner: 10Hashar) [12:26:18] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [13:03:36] !log ores:7b987a7 is going beta [13:03:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:10:10] RECOVERY - Puppet errors on deployment-certcentral-testdns is OK: OK: Less than 1.00% above the threshold [0.0] [13:21:42] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:22:28] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:17:18] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [14:22:19] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [14:29:58] 10Phabricator, 10Cleanup: Form 33 amendment request - https://phabricator.wikimedia.org/T204487 (10Dinoguy1000) Thanks! [15:13:42] !log launching integration-slave-docker-1037 bigram instance [15:13:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:15:54] !log taking integration-slave-docker-1036 offline due to unusually high cpu steal % trend [15:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:49] !log integration-slave-docker-1035/1036 showing unusually high cpu steal and unusually long mean build durations [15:17:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:21:06] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:28:18] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [15:33:18] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [15:33:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: maintenance-disconnect-full-disks gets stuck - https://phabricator.wikimedia.org/T204077 (10thcipriani) 05Open>03Resolved https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/4319/conso... [15:37:13] no_justification https://gerrit-review.googlesource.com/c/gerrit/+/196790 [15:41:14] 10Scap, 10Operations, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10thcipriani) >>! In T202100#4543309, @Ottomata wrote: > Done, worked everywhere except: > > ``` > The following hosts were unreachable: > cloudservices1004.wikimedia.org > ```... [15:42:46] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:48:15] !log taking node integration-slave-docker-1035 offline due to unusually high steal cpu time and long build durations [15:48:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:49:46] !log replacing integration-slave-docker-1036 with new bigram instance [15:49:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:53:56] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [16:17:45] PROBLEM - Puppet errors on integration-slave-docker-1037 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:22:48] RECOVERY - Puppet errors on integration-slave-docker-1037 is OK: OK: Less than 1.00% above the threshold [0.0] [16:37:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: logstash-beta.wmflab throws multiple "Error: Could not locate that visualization" - https://phabricator.wikimedia.org/T204845 (10Etonkovidova) a:03fgiunchedi [16:46:28] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Operations, 10Wikimedia-Logstash: logstash-beta.wmflab throws multiple "Error: Could not locate that visualization" - https://phabricator.wikimedia.org/T204845 (10greg) @fgiunchedi fyi, you got assigned to this task because... [16:47:43] !log added new jenkins node integration-slave-docker-1037 with 7 executors [16:47:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:51:37] PROBLEM - Host integration-slave-docker-1035 is DOWN: CRITICAL - Host Unreachable (10.68.21.100) [17:07:02] (03PS1) 10Hashar: Decouple composer/npm run for extskin [integration/quibble] - 10https://gerrit.wikimedia.org/r/461998 [17:07:04] (03PS1) 10Hashar: Early return in extskin composer/npm test commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/461999 [17:09:18] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [17:12:24] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:17] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [17:17:25] (03CR) 10jerkins-bot: [V: 04-1] Decouple composer/npm run for extskin [integration/quibble] - 10https://gerrit.wikimedia.org/r/461998 (owner: 10Hashar) [17:20:16] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [17:26:11] !log adding jenkins node integration-slave-docker-1038 with 7 executors [17:26:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:30:20] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [17:32:21] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [17:43:24] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:20] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [17:57:27] PROBLEM - Puppet errors on deployment-mwmaint01 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [17:58:15] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [18:07:27] RECOVERY - Puppet errors on deployment-mwmaint01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:23:26] PROBLEM - Puppet errors on deployment-mwmaint01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:28:50] PROBLEM - Puppet errors on deployment-mwmaint01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:30:12] ^ me fiddling [18:33:28] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics: Add Code Health Metrics Working Group members to Code Health mailing list - https://phabricator.wikimedia.org/T205132 (10Jrbranaa) [19:07:16] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [19:48:27] RECOVERY - Puppet errors on deployment-mwmaint01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:18:58] 10Beta-Cluster-Infrastructure, 10Operations, 10Wikidata, 10wikidata-tech-focus, and 3 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10thcipriani) I've got a stretch instance called `deployment-mwmaint01` running in beta with `role::mediawiki_mainten... [20:43:41] 10Gerrit, 10Developer-Advocacy, 10GitHub-Mirrors, 10Repository-Admins, 10User-Tgr: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540 (10D3r1ck01) [20:44:20] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [20:49:17] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [20:55:16] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [21:05:17] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [21:07:51] 10Gerrit, 10Developer-Advocacy, 10GitHub-Mirrors, 10Repository-Admins, 10User-Tgr: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540 (10hashar) 05Open>03Resolved a:03hashar The Code of Conduct is already listed everywhere (eg Phabricator, soon Gerrit etc).... [21:11:19] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [21:12:21] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:13:52] 10Beta-Cluster-Infrastructure, 10media-storage: On beta enable swift statsd metric - https://phabricator.wikimedia.org/T161084 (10hashar) 05Open>03declined We don't really monitor Swift on beta. [21:13:55] 10Beta-Cluster-Infrastructure, 10RelEng-Archive-FY201718-Q1, 10media-storage, 10Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990 (10hashar) [21:16:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday): On contint-operations-puppet apt-get yields about locale - https://phabricator.wikimedia.org/T174584 (10hashar) 05Open>03declined [21:19:58] 10Deployments, 10Release-Engineering-Team (Someday), 10Scap, 10scap2: scap3 allows plugins to declare dependencies we do not support - https://phabricator.wikimedia.org/T154482 (10hashar) 05Open>03Invalid [21:22:25] 10Continuous-Integration-Infrastructure: Use dedicated jobs for mediawiki-core wmf branches that honour submodules - https://phabricator.wikimedia.org/T88239 (10hashar) 05Open>03declined After some years, I don't think it will serve much purpose. Some mediawiki extensions are incompatible with each other, s... [21:29:47] 10Beta-Cluster-Infrastructure, 10monitoring, 10Tracking: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497 (10hashar) [21:29:49] 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 10monitoring: Beta cluster job queue is unmonitored / potentially not running all jobs - https://phabricator.wikimedia.org/T128357 (10hashar) 05stalled>03Invalid That is no more accurate. The background jobs processing has been overhauled meanwhile. [21:37:29] 10Phabricator, 10Cleanup: Form 33 amendment request - https://phabricator.wikimedia.org/T204487 (10MarcoAurelio) Thanks indeed. [21:40:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: maintenance-disconnect-full-disks gets stuck - https://phabricator.wikimedia.org/T204077 (10hashar) Timeout wrapping solved it I guess. Thank you! [21:41:34] 10Beta-Cluster-Infrastructure, 10Operations, 10Wikidata, 10wikidata-tech-focus, and 3 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10MarcoAurelio) Note that not all scripts running on production can be run on beta (for example, the lack of CheckUse... [21:43:27] 10Beta-Cluster-Infrastructure: Generate beta captchas - https://phabricator.wikimedia.org/T204611 (10MarcoAurelio) >>! In T204611#4604565, @Krenair wrote: > Okay so it's fine with 952. It'll give you one swift error for every one over that. Thanks for regenerating them although it looks kinda strange that it le... [21:47:21] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:01:41] 10Beta-Cluster-Infrastructure, 10Operations, 10Wikidata, 10wikidata-tech-focus, and 3 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10Dzahn) Currently there is just one general switch to either enable all crons (scripts) or disable them all, and it'... [22:32:12] 10Phabricator, 10User-MModell: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) >>! In T93499#4605241, @MGChecker wrote: > Couldn't this problem be circumvented by just allowing to add a subtype (e.g. changing it from the default?) This should satisfy the main use case without... [22:36:57] 10Beta-Cluster-Infrastructure: Generate beta captchas - https://phabricator.wikimedia.org/T204611 (10Krenair) no [23:32:11] 10Scap, 10Operations, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10Dzahn) ``` [cloudservices1004:~] $ dpkg -l git-fat .. ii git-fat 0.1.3-2~jessie1 all Manage large files with git, without checking ``` [23:33:59] 10Scap, 10Operations, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10Dzahn) 05Open>03Resolved a:03Ottomata