[00:04:34] So I tried that captcha thing earlier [00:04:50] just gets weirder [00:05:15] https://phabricator.wikimedia.org/P7572 [00:08:54] That host lookup is a filed bug [00:09:03] Fatal error: require(/srv/mediawiki/php-master/includes/json/FormatJson.php): File not found in /srv/mediawiki/php-master/includes/AutoLoader.php on line 109 [00:09:04] That's lol [00:09:17] I don't quite believe it [00:11:13] Doesn't even use it... [00:38:52] Reedy, might be for logging? [00:43:25] Is MWScript needed for GenerateFancyCaptchas.php ? [00:44:48] hmm [00:44:55] what's the beta bastion? [00:46:56] Platonides, the beta bastion? [00:47:00] the standard labs bastion [00:47:04] bastion.wmflabs.org [00:47:12] I'm in [00:47:29] trying that same command [00:54:07] this script would benefit of some more parallelization btw [00:56:08] let's not run before we can walk Platonides [00:56:13] xD [00:56:54] it seems it already does that with multiple threds :P [00:57:21] *threads [00:57:22] --threads", help="Maximum number of threads to be used to generate captchas.", type='int', default=1 [01:16:18] Hm.. scap is checking logstash of canary hosts in eqiad only during deployment [01:16:22] that seems incorrect [01:16:33] given they receive no traffic [01:20:41] 10Release-Engineering-Team, 10Scap: Scap is checking canary servers in dormant instead of active-dc - https://phabricator.wikimedia.org/T204907 (10Krinkle) [01:21:18] 10Release-Engineering-Team, 10Scap, 10Datacenter-Switchover-2018: Scap is checking canary servers in dormant instead of active-dc - https://phabricator.wikimedia.org/T204907 (10Krinkle) [01:23:05] Project beta-scap-eqiad build #222472: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222472/ [01:23:07] 10Continuous-Integration-Config, 10MinervaNeue, 10phan-taint-check-plugin, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1): Minerva taint error: Calling method \BaseTemplate::set() in \SkinMinerva::prepareHeaderAndFooter that outputs using tainted ar... - https://phabricator.wikimedia.org/T204094 [01:23:07] Project beta-update-databases-eqiad build #28448: 04FAILURE in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28448/ [01:23:55] Project beta-scap-eqiad build #222473: 04STILL FAILING in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222473/ [01:44:22] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T191068 (10Krinkle) [01:45:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T191068 (10Krinkle) [01:47:09] Yippee, build fixed! [01:47:10] Project beta-scap-eqiad build #222474: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222474/ [02:17:16] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) WARN: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<100.00%) [02:22:42] Yippee, build fixed! [02:22:42] Project beta-update-databases-eqiad build #28449: 09FIXED in 2 min 41 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28449/ [02:52:18] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<11.11%) [03:00:55] James_F: Hm. did the wmf branch phab tag milestone names get longer? [03:01:12] It's not fitting on one line anymore when editing projects list on a task [03:01:14] > MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)) [03:01:55] 2x WMF, 2x 1.32; could maybe lose one level of parenthesis and redundancy, but not sure who to propose to or where the change would be made. [03:03:14] e.g. MW-1.32-release-notes (1.32.0-wmf.22; Deploy 2018-09-18) [03:03:33] They didn't get any longer. [03:03:54] But it's entirely possible that a software change in Phabricator upstream got deployed that changes margins/text sizes. [03:04:38] I'm not sure I like "MW-1.32-release-notes (1.32.0-wmf.22; Deploy 2018-09-18)" [03:05:08] If I was going for brevity, I'd pick "MW-1.32-notes (1.32.0-wmf.22; 2018.09.18)" [03:06:48] Yeah, I wasn't sure whether to keep "deploy" just figured since it was there already [03:06:59] Hm.. German date notation? [03:07:44] * Krinkle approved :D [03:07:48] * Krinkle approves * [03:09:06] I also like the dropping of 'release' (from the primary form at least) to avoid autocomplete-fighting against actual release [03:11:31] - is wider than . [03:11:53] Ideally I'd go with old British notation and write 2018.ix.18 but most humans would complain. :-) [03:12:07] https://usercontent.irccloud-cdn.com/file/uJQunxyb/Screen%20Shot%202018-09-20%20at%2004.11.49.png [03:12:18] (console editing, not real) [03:12:25] Certainly, it's shorter. [03:12:51] File a Phab task where people can bikeshed and then I'll implement next week? [03:12:55] Anyhow, either form is fine. I'll leave with you. [03:13:02] OK. [03:19:25] James_F: Sir, https://phabricator.wikimedia.org/T204909 [03:22:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) WARN: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<100.00%) [03:42:16] Thanks, Krinkle. [04:50:36] (03PS1) 10Brian Wolff: Make MinervaNeue run seccheck non-voting. [integration/config] - 10https://gerrit.wikimedia.org/r/461510 (https://phabricator.wikimedia.org/T204094) [04:52:24] (03CR) 10Brian Wolff: "Please note, I'm not very familiar with how integration stuff works, and I don't really know how to test this locally, so I may have made " [integration/config] - 10https://gerrit.wikimedia.org/r/461510 (https://phabricator.wikimedia.org/T204094) (owner: 10Brian Wolff) [05:47:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) WARN: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<100.00%) [05:56:41] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [06:27:10] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:48:21] 10Project-Admins: Create Federated-Wikibase-Workshops@NewYork-Rhizome-workshop project - https://phabricator.wikimedia.org/T204913 (10Addshore) [07:07:11] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:42:19] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) WARN: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<100.00%) [07:47:12] 10Project-Admins: Create phabricator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10Aklapper) [07:47:22] 10Project-Admins: Create phabricator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10Aklapper) 05Open>03stalled [07:57:16] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) WARN: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<100.00%) [07:57:27] 10Project-Admins: Create Federated-Wikibase-Workshops@NewYork-Rhizome-workshop project - https://phabricator.wikimedia.org/T204913 (10Aklapper) @Addshore: I've given you project creation permissions. Please use them wisely™ and [[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Types_of_Projects |... [07:59:24] 10Project-Admins, 10User-Addshore: Create Federated-Wikibase-Workshops@NewYork-Rhizome-workshop project - https://phabricator.wikimedia.org/T204913 (10Addshore) 05Open>03Resolved a:03Addshore Thanks @Aklapper. Created @ https://phabricator.wikimedia.org/project/view/3614/ [08:27:19] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) WARN: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<100.00%) [08:43:33] 10Continuous-Integration-Config, 10Operations-Software-Development: tox-docker fails on keyholder due to pycrypto - https://phabricator.wikimedia.org/T204919 (10hashar) [08:50:58] 10Project-Admins: Create phabricator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10ArielGlenn) Hey @Aklapper, the bot is on toolforge though not currently running. Is that what you're asking? [09:00:41] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Quibble, 10User-zeljkofilipin: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10zeljkofilipin) >>! In T199116#4597824, @phuedx wrote: > W... [09:04:21] 10Continuous-Integration-Config, 10Operations-Software-Development: tox-docker fails on keyholder due to pycrypto - https://phabricator.wikimedia.org/T204919 (10hashar) At least the pycrypto install works: ``` $ docker pull docker-registry.wikimedia.org/releng/tox:0.1.1 $ docker run --rm -it --entrypoint=/bin/... [09:13:02] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Mz83ude) Thank you for the clean up and the tips. :) [09:20:19] 10Project-Admins, 10Security-Team: Undefined #Security-General and #Security-Other - https://phabricator.wikimedia.org/T109328 (10Aklapper) >>! In T109328#4569983, @Tgr wrote: > IMO, open tasks should be added to #security and closed ones are not worth worrying about. Right. Done for those public ones by usin... [09:23:42] 10Release-Engineering-Team, 10Cloud-VPS, 10Operations, 10puppet-compiler, and 3 others: Upgrade Puppet compilers to Stretch - https://phabricator.wikimedia.org/T191438 (10aborrero) [09:29:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Evaluate different strategy for Docker CI instances - https://phabricator.wikimedia.org/T202160 (10aborrero) [09:29:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Request increased quota for integration Cloud VPS project - https://phabricator.wikimedia.org/T204373 (10aborrero) 05Open>03Resolved a:03aborrero Done! For the record, this is what I did: ```lines=20 ab... [09:30:49] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) @Mz83ude: Would it be possible to add a `license-name` field/[[ https://spdx.org/licenses/ | value ]] to `extension.json` and the infobox on https://www.mediawiki.org/wiki/Extension:ChangeUserPasswords ? [09:34:13] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) Requested public project #MediaWiki-extensions-ChangeUserPasswords has been created: https://phabricator.wikimedia.org/project/view/3615/ Please encourage interested people to visit the project and to j... [09:34:20] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) 05Open>03Resolved [09:36:00] Hi! Not sure if this is the right channel to ask, but if not maybe you can send me elsewhere [09:36:30] Is there a way to set up Phabricator to mail any changes / additions tagged as a specific project and all subprojects to a mailing list? [09:36:50] #wikimedia-devtools is best [09:37:12] I went there, but the topic said it's closed and to come here? :) [09:37:20] Ah, heh. Good one. :) [09:37:20] In theory you can create a Phab account with that email address assigned to that account, and then make that account watch that project [09:37:40] (it's us at Wikimedia Eesti again, trying to see what we can do with https://phabricator.wikimedia.org/project/subprojects/3560/:) ) [09:37:51] That's the setup for https://phabricator.wikimedia.org/p/Pywikibugs/ for example [09:38:36] Whoops, I'm wrong, it's https://phabricator.wikimedia.org/p/pywikibot-bugs-list/ [09:38:39] some email systems can also forward automatically based on a filter, I believe [09:38:55] which is implemented via a filter rule at https://phabricator.wikimedia.org/H6 - see https://www.mediawiki.org/wiki/Phabricator/Help/Herald_Rules [09:42:27] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:42:38] I can't see https://phabricator.wikimedia.org/H6 but from what I understand I can create my own rules, right? I'll give that a go and ask if I get stuck then :) [09:42:49] oops [09:43:20] it's a global rule to "Add subscribers: pywikibot-bugs-list" if project is X or project is Y [09:43:56] Oh, I also can't see https://phabricator.wikimedia.org/herald/ :) [09:44:33] So I guess I can't create a rule either :D [09:44:36] meh. https://www.mediawiki.org/wiki/Phabricator/Help/Herald_Rules describes the interface, with a screenshot of available option and how to request a rule [09:45:33] So I guess I can't create it myself but I can request it via Phab task? Ok, will do that [09:45:35] Thanks! [09:46:59] yes [09:47:00] thanks [09:47:16] we'll probably need to know a Phab account linked to that mailing list first, though [09:47:46] Sure, got that much :) So, create account, create task saying "please add this account as a subscriber for project X and subprojects" [09:47:59] Will do that, no probs. Thanks again [09:48:24] reosarevok: ....and creating that Phab account requires being connected to a mediawiki.org or LDAP account, and I can imagine that it might send the email with your password to the mailing list. Just saying. So you may want to change the password quickly **if** it's a public mailing list. [09:49:08] It'll be a new mailing list, so it should be fine :) But yeah, good point, will make sure that's gone before I make anything public [10:11:05] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Quibble, 10User-zeljkofilipin: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10hashar) The logic is #quibble `quibble/cmd.py`, roughly w... [10:11:14] zeljkof: https://phabricator.wikimedia.org/T199116#4601051 [10:11:35] zeljkof: I might have a solution to not have to run npm install twice :] [10:11:40] E next week [10:11:45] poke me again about it on monday [10:14:58] :) [10:15:02] Will do [10:17:25] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [10:21:29] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Mz83ude) >>! In T202275#4600935, @Aklapper wrote: > @Mz83ude: Would it be possible to add a `license-name` field/[[ https://spdx.org/licenses/ | value ]] to `extension.json` and the infobox on https://www.mediawik... [10:22:50] 10Project-Admins: Create phabricator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10Aklapper) a:03Aklapper Yes, thanks. Looks like https://tools.wmflabs.org/wm-commons-emoji-bot/ and https://tools.wmflabs.org/admin/tool/wm-commons-emoji-bot , for the project description. [10:23:10] Project beta-scap-eqiad build #222509: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222509/ [10:23:11] Project beta-update-databases-eqiad build #28457: 04FAILURE in 3 min 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28457/ [10:27:17] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<55.56%) WARN: deployment-prep.deployment-deploy01.diskspace._srv.byte_percentfree (<100.00%) [10:28:37] 10Project-Admins: Create phabricator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10Aklapper) 05stalled>03Resolved Requested public project #commons-twitter-bot has been created: https://phabricator.wikimedia.org/project/view/3616/ I kept the name consistent with the Gerrit repo... [10:31:29] 10Phabricator, 10Cleanup: Form 33 amendment request - https://phabricator.wikimedia.org/T204487 (10Aklapper) 05Open>03Resolved a:03Aklapper Updated. Previous "Preamble" content on https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/edit/33/ : ``` Use this form for requesting an ext... [10:36:34] Project beta-scap-eqiad build #222510: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222510/ [10:38:05] 10Project-Admins, 10wikiba.se: permit a phabricator page for the FactGrid project - https://phabricator.wikimedia.org/T193071 (10Aklapper) @Olaf_Simons: Any news? [10:41:33] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) @Mz83ude: `CC-BY-NC-4.0 ` is a non-free license and (in my understanding) violates the requirements to be hosted in Wikimedia Gerrit. See https://www.mediawiki.org/wiki/Gerrit/New_repositories [10:43:28] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) Also note https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software [10:49:26] Project beta-scap-eqiad build #222511: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222511/ [11:01:34] 10Continuous-Integration-Config, 10Operations-Software-Development: tox-docker fails on keyholder due to pycrypto - https://phabricator.wikimedia.org/T204919 (10hashar) I could not reproduce the issue so I went with live debugging in attempt to get the autoconf log file which is somewhere under /tmp. To do so... [11:02:31] Project beta-scap-eqiad build #222512: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222512/ [11:07:45] !log deployment-deploy01 is out of disk space (again) [11:07:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:09:40] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 10Discovery-Search: Wikibase related test errors for CirrusSearch blocking CX CI - https://phabricator.wikimedia.org/T204933 (10santhosh) [11:10:21] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:10:26] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-WikibaseMediaInfo: Wikibase related test errors for CirrusSearch blocking CX CI - https://phabricator.wikimedia.org/T204933 (10santhosh) [11:10:50] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Mz83ude) >>! In T202275#4601127, @Aklapper wrote: > Also note https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software Corrected. Now it has GPL-2.0-or-later. [11:17:05] !log deployment-deploy01: removed /srv/deployment/analytics/refinery-cache (8GBytes) [11:17:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:17:15] Yippee, build fixed! [11:17:15] Project beta-scap-eqiad build #222513: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/222513/ [11:18:26] 10Continuous-Integration-Config, 10Operations-Software-Development: tox-docker fails on keyholder due to pycrypto - https://phabricator.wikimedia.org/T204919 (10hashar) The `--tmpfs /tmp` comes from abandoned change https://gerrit.wikimedia.org/r/#/c/integration/config/+/457070/ we did for T203181 [11:20:14] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10Regression: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 (10hashar) I have abandoned the change https://gerrit.wikimedia.org/r/#/c/integration/c... [11:21:21] !log Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. | T203181 and T204919 [11:21:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:21:29] T203181: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 [11:21:29] T204919: tox-docker fails on keyholder due to pycrypto - https://phabricator.wikimedia.org/T204919 [11:22:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10Regression: Quibble MariaDB should use a tmpfs as a datadir - https://phabricator.wikimedia.org/T203181 (10hashar) contint1001$ grep -l '\--tmpfs /tmp' /var/lib/jenkins/jobs/*/config.xml|cut -... [11:22:40] Yippee, build fixed! [11:22:40] Project beta-update-databases-eqiad build #28458: 09FIXED in 2 min 39 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28458/ [11:25:58] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-WikibaseMediaInfo: Wikibase related test errors for CirrusSearch blocking CX CI - https://phabricator.wikimedia.org/T204933 (10dcausse) [11:27:16] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [11:27:43] 10Continuous-Integration-Config, 10Operations-Software-Development: tox-docker fails on keyholder due to pycrypto - https://phabricator.wikimedia.org/T204919 (10hashar) 05Open>03Resolved a:03hashar `/tmp` was mounted as a tmpfs which comes with the `noexec` flag. I guess autoconf compiles a binary and th... [11:35:37] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-WikibaseMediaInfo: Wikibase related test errors for CirrusSearch blocking CX CI - https://phabricator.wikimedia.org/T204933 (10dcausse) a:03Cparle Should be fixed by https://gerrit.wikimedia.org/r/c/mediaw... [11:45:22] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:36] 10Continuous-Integration-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-WikibaseMediaInfo: Wikibase related test errors for CirrusSearch blocking CX CI - https://phabricator.wikimedia.org/T204933 (10dcausse) 05Open>03Resolved Fixed, sorry about that and thanks for the @Cparle! [12:12:48] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [12:46:42] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:00:51] 10Phabricator: Herald rule: subscribe WMEE-Phabricator-List to tasks in WMEE (sub)project(s) - https://phabricator.wikimedia.org/T204946 (10reosarevok) [13:00:53] 10Project-Admins: Requesting personal project tag User-banyek - https://phabricator.wikimedia.org/T204297 (10Banyek) Thank you! [13:12:26] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:24:31] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1): exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Ottomata) @Krenair oh ya?... [13:43:14] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q1): exported puppet resources are not queryable: cannot create grafana graphs of EventLogging running in beta cluster - https://phabricator.wikimedia.org/T204088 (10Krenair) I think so - I'm p... [14:06:58] 10Phabricator: Herald rule: subscribe WMEE-Phabricator-List to tasks in WMEE (sub)project(s) - https://phabricator.wikimedia.org/T204946 (10reosarevok) I've seen a suggestion in T108586 that this could be done in a better way by having the user watch the project(s). If this would work and is considered better, d... [14:26:17] (03CR) 10Jdlrobson: [C: 031] Make MinervaNeue run seccheck non-voting. [integration/config] - 10https://gerrit.wikimedia.org/r/461510 (https://phabricator.wikimedia.org/T204094) (owner: 10Brian Wolff) [14:29:46] 10Release-Engineering-Team, 10Scap, 10Datacenter-Switchover-2018, 10Patch-For-Review: Scap is checking canary servers in dormant instead of active-dc - https://phabricator.wikimedia.org/T204907 (10hashar) After discussion with Alexandros and Giuseppe, for now we have just updated the list of hosts in the d... [14:35:57] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994 (10zeljkofilipin) [14:36:00] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10zeljkofilipin) [14:36:03] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Patches in Gerrit deleting Selenium+Ruby tests for repositories that still have them - https://phabricator.wikimedia.org/T183162 (10zeljkofilipin) 05Open>03declined I don't think this is a good idea any more. Until Seleniu... [14:44:19] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:44:22] 10Release-Engineering-Team, 10Wikimedia-production-error: Promoting group1 to 1.32.0-wmf.22 caused a spam of web request took longer than 60 seconds and timed out - https://phabricator.wikimedia.org/T204871 (10hashar) Same happened with group2: 14:38:57 Finished sync-apaches (duration: 00m 38s) {F26103910 s... [14:48:20] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:50:14] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:54:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T191068 (10hashar) [14:54:55] 10Project-Admins: Create phabricator project for Emoji-TwitterBot - https://phabricator.wikimedia.org/T204301 (10rosalieper) Thanks! [14:56:45] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [14:56:58] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) a:03zeljkofilipin [15:02:04] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:04:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:07:40] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:14:32] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:22:02] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:26:28] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:27:10] 10Beta-Cluster-Infrastructure: Generate beta captchas - https://phabricator.wikimedia.org/T204611 (10Krenair) Yeah so this is not going as intended: {P7572} [15:34:25] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:39:57] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [15:40:43] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [16:03:04] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [16:11:26] RelEnger bikeshedding input on T204909 very welcome. [16:11:27] T204909: Reduce length of Phabricator milestone names for wmf-branches - https://phabricator.wikimedia.org/T204909 [16:21:27] (03CR) 10Hashar: [C: 032] Archive several OfflineContentGenerator/* repositories [integration/config] - 10https://gerrit.wikimedia.org/r/461477 (https://phabricator.wikimedia.org/T183891) (owner: 10MarcoAurelio) [16:23:05] (03Merged) 10jenkins-bot: Archive several OfflineContentGenerator/* repositories [integration/config] - 10https://gerrit.wikimedia.org/r/461477 (https://phabricator.wikimedia.org/T183891) (owner: 10MarcoAurelio) [16:24:45] (03PS11) 10Legoktm: Move to `git archive` like model for MediaWiki releases [tools/release] - 10https://gerrit.wikimedia.org/r/454609 (https://phabricator.wikimedia.org/T199467) [16:35:28] (03CR) 10Hashar: [C: 032] Mark repository as read only [services/ocg-collection] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/461475 (https://phabricator.wikimedia.org/T183891) (owner: 10MarcoAurelio) [16:52:08] !log deploy ores:ee2d28b [16:52:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:00:39] just a complaint, every other oid is two syllables: mathoid graphoid parsoid [17:00:44] so really it should be blubboid [17:00:46] just sayin [17:08:38] apergos: :P [17:08:53] :-D [17:10:35] but ..but .. the name is composed by adding the -oid suffix to the service name.. consistency :P [17:11:07] maybe settle on "blubbroid" [17:24:06] 10Phabricator, 10Performance: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10mmodell) We are now at ~300ms [17:28:45] 10Phabricator, 10Performance: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10Jdforrester-WMF) [17:30:09] nope. parsoid is the name of the service. but graph is the name of that service. [17:30:16] naming is hard [17:41:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) [17:46:02] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5) - https://phabricator.wikimedia.org/T191921 (10thcipriani) 05Open>03Resolved a:03thcipriani I ran a no-op `scap sync` using the patch... [17:47:49] 10Phabricator, 10Performance: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10Quiddity) Is it the number of //Rules// that is the problem, or more the number of //Conditions//? I.e. would it help if we asked users who have 2 rules that could easily... [17:52:37] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5) - https://phabricator.wikimedia.org/T191921 (10Legoktm) Yay! :-) [18:01:49] (03PS12) 10Reedy: Move to `git archive` like model for MediaWiki releases [tools/release] - 10https://gerrit.wikimedia.org/r/454609 (https://phabricator.wikimedia.org/T199467) (owner: 10Legoktm) [18:01:58] (03CR) 10Reedy: [C: 032] "<3" [tools/release] - 10https://gerrit.wikimedia.org/r/454609 (https://phabricator.wikimedia.org/T199467) (owner: 10Legoktm) [18:02:39] (03Merged) 10jenkins-bot: Move to `git archive` like model for MediaWiki releases [tools/release] - 10https://gerrit.wikimedia.org/r/454609 (https://phabricator.wikimedia.org/T199467) (owner: 10Legoktm) [18:02:57] whee [18:10:00] 10MediaWiki-Releasing, 10Release-Engineering-Team: Signing MW Core Tags - https://phabricator.wikimedia.org/T204995 (10Reedy) [18:10:36] https://github.com/susu9/pick-patch looks potentially interesting for people that want to grab a number of gerrit patches and sequence them on top of one another. [18:11:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Request increased quota for integration Cloud VPS project - https://phabricator.wikimedia.org/T204373 (10dduvall) Thank you! [18:11:12] 10MediaWiki-Releasing: Move to a `git archive` like model for MediaWiki tarball creation - https://phabricator.wikimedia.org/T199467 (10Jdforrester-WMF) [18:13:00] (03PS1) 10Reedy: DING DING DING [tools/release] - 10https://gerrit.wikimedia.org/r/461701 [18:13:28] (03PS2) 10Reedy: DING DING DING [tools/release] - 10https://gerrit.wikimedia.org/r/461701 [18:14:38] What does that do Reedy? [18:15:05] (03PS3) 10Legoktm: DING DING DING when it's time to GPG sign tarballs [tools/release] - 10https://gerrit.wikimedia.org/r/461701 (owner: 10Reedy) [18:15:07] Terminal bell [18:15:13] (03CR) 10Legoktm: [C: 032] DING DING DING when it's time to GPG sign tarballs [tools/release] - 10https://gerrit.wikimedia.org/r/461701 (owner: 10Reedy) [18:15:13] ie gets your attention [18:15:37] because if you don't enter a password on gpg fast enough, it'll time out and fail [18:15:39] doesn't seem to do anything on my laptop :/ [18:15:46] (03Merged) 10jenkins-bot: DING DING DING when it's time to GPG sign tarballs [tools/release] - 10https://gerrit.wikimedia.org/r/461701 (owner: 10Reedy) [18:15:52] Windows? [18:16:01] no [18:16:05] ubuntu 18.04 [18:16:41] Definitely bleeps from my 18.04 vm through ssh to my mac [18:17:32] 10Phabricator, 10Performance: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10Aklapper) I see some patterns, like * "inform me when natural language foo was mentioned" (such as H188, H92, H81), * "email me explicitly when tasks in project XYZ get t... [18:17:34] "The reason this doesn't is because by default Ubuntu no longer loads the hardware driver that produce beeps." [18:20:09] andre__: Maybe we should make each user of Herald pay $1k a year for the privilege? It'd reduce the load substantially. :-) [18:21:24] Reedy, :( [18:21:30] (03PS2) 10Jforrester: Order list of extensions by alphabet [tools/release] - 10https://gerrit.wikimedia.org/r/455186 (owner: 10Aklapper) [18:23:31] those pings remind me of the TCAS sometimes [18:26:03] Traffic Collsion and Avoidance Service? [18:26:26] Reedy: yep [18:31:28] although iirc it says "traffic" before beeping [18:32:08] It's a bit more than beeping [18:32:27] a friend of mine is a pilot and he took me sometimes to airbus simulators [18:34:32] is this the thing that yells a word I can't repeat here at the pilot? [18:35:30] RETARD [18:35:32] That one? [18:35:36] No, that's landing stuff :P [18:35:41] aw [18:35:47] and TERRAIN [18:36:08] * Hauskatze likes ILS landings [18:36:10] well, terrain isn't offensive [18:36:39] James_F, disabling personal rules is easier :P [18:36:51] Especially as there are some that are bloody pointles [18:36:52] s [18:37:01] andre__: That too. [18:37:13] Can't remember why we allowed that. [18:37:23] Reedy: You're a damn TERRAIN! ;-) [18:37:34] TERRAIN TERRAIN PULL UP WOOP WOOP TERRAIN [18:40:19] you should make jouncebot use some of those terms for deploys [18:40:31] train -> plane [18:40:59] andre__: please replace my personal herald rule with a non-personal rule ?:) [18:41:07] ding dong Dear pasangers please be adviced fligh LH454 is delayed due to MediaWiki train mutante [18:41:37] i was more thinking of the "TERRAIN TERRAIN PULL UP" part when there are 5xx alerts after deploy [18:43:37] or the crash landing one mutante [18:43:55] heh [18:45:02] SINK RATE PULL UP [18:49:55] Ugh, back on topic for a bit... [18:50:05] greg-g: There's far too many versions of MW on deploy1001 atm [18:50:24] reedy@deploy1001:/srv/mediawiki-staging$ ls -al | grep -c php\-1 [18:50:25] 17 [18:50:30] Not checked how many are just assets etc [18:50:39] But we definitely don't need them all [18:53:53] 10Release-Engineering-Team, 10Wikimedia-General-or-Unknown: Too many versions of MW core on deploy1001 - https://phabricator.wikimedia.org/T205000 (10Reedy) [18:54:04] 10Release-Engineering-Team, 10Wikimedia-General-or-Unknown: Too many versions of MW core on deploy1001 - https://phabricator.wikimedia.org/T205000 (10Reedy) p:05Triage>03High [18:59:20] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday): Removal of EOL release branches broke SemanticMediaWiki CI - https://phabricator.wikimedia.org/T179754 (10greg) >>! In T179754#3752576, @greg wrote: >>>! In T179754#3752248, @Legoktm wrote: >> I was thinking more like a year or two after EOL. > > Th... [19:00:07] Reedy: I thought we had a task about automating the removal of those old branches... [19:00:12] there's scap clean, I believe [19:00:17] Yeah [19:00:31] I was gonna kill some, but noticed Tyler was fixing an oopsie from swat :) [19:00:48] It's the train window, so a good time to do so [19:09:56] 10Project-Admins: Create User-Daimona project - https://phabricator.wikimedia.org/T204407 (10MGChecker) [19:10:08] 10Phabricator (Upstream), 10Upstream: Per-user projects for personal work in progress tracking - https://phabricator.wikimedia.org/T555 (10MGChecker) [19:15:28] greg-g: Oh, not as bad as it looked [19:15:31] most are just empty folders [19:16:50] 10Release-Engineering-Team, 10Wikimedia-General-or-Unknown: Too many versions of MW core on deploy1001 - https://phabricator.wikimedia.org/T205000 (10Reedy) 05Open>03Invalid Most are just empty folders... Will file a task for that ``` lrwxrwxrwx 1 hashar wikidev 17 Sep 19 19:28 php -> php-1.3... [19:16:58] Do we even need .14/.15/.16? [19:17:40] mmm, probably not [19:17:44] need to check dates [19:18:17] 10Scap: scap clean not cleaning up fully - https://phabricator.wikimedia.org/T205002 (10Reedy) [19:19:05] wmf.18 was by definition a month ago now. [19:19:19] Well, four weeks and three days. [19:20:57] What's the cut off? [19:23:00] I have no idea. We stop making the CDB files at some point pretty quickly, but presumably we keep the static assets for longer? [19:23:44] `du -sh php-1.32.0-wmf.*` is fun. [19:31:39] 10Scap: prune-git-branches doing extra branches - https://phabricator.wikimedia.org/T205006 (10Reedy) [19:35:23] James_F: https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Clean_up_old_stuff [19:35:29] "For all branches more than 30 days old, drop everything." [19:36:13] Ah, thanks. [19:36:21] I can never remeber either [19:36:27] neither can I remember [19:36:35] For some "strange" reason I've never read the Train deploys documentation too closely. ;-) [19:37:16] choo choo [19:37:23] reedy@deploy1001:/srv/mediawiki-staging$ find . -maxdepth 1 -type d -name 'php-*' -ctime +30 [19:37:23] reedy@deploy1001:/srv/mediawiki-staging$ [19:37:27] Looks like we're up to date now [19:37:33] Yay. [19:37:47] And a few GiB saved on each host. [19:38:10] And some time saved scapping I guess [19:39:47] The HHVM -> PHP7 scap change has saved ~40mins per full scap, it seems, so this is just icing on that cake. ;-) [19:41:29] I for one welcome our fast deploying future [19:41:55] Next thing you know we'll be able to merge and deploy something in less than 30 minutes! ;-) [19:43:05] jerkins will keep us in reality [19:45:48] Scap used to take like 30 minutes total didn't it? [19:46:24] hm apparently not [19:46:43] It's taken varying amounts of time :) [19:47:05] I believe in a magical future where deploying builds a new docker image, injects it into the cluster, and requests are atomically switched over with checks as the load expands. [19:47:15] What facebook crack have you been smoking? [19:48:17] * James_F grins. [19:48:20] lol [19:49:08] Looking at a few archive SAL entries, ~12–15 mins was common only a few years ago. [19:49:34] a few years ago there was no codfw [19:50:01] I cheated and picked a time when there was pmtpa and eqiad. [19:57:54] (03PS1) 10Reedy: Stop branching DisableAccount [tools/release] - 10https://gerrit.wikimedia.org/r/461733 (https://phabricator.wikimedia.org/T106067) [19:59:19] 10MediaWiki-Releasing, 10Release-Engineering-Team (Someday): Removal of EOL release branches broke SemanticMediaWiki CI - https://phabricator.wikimedia.org/T179754 (10Aklapper) >>! In T179754#4603336, @greg wrote: > Where should we document this? Closest I can come up with is https://www.mediawiki.org/wiki/Re... [20:06:39] thcipriani: do the swagger checks run before or after code is serving prod traffic on canaries? [20:07:06] I know the logstash checks don't auto-revert, but the swagger checks can run before that, right? [20:07:36] Swagger checks happen right after canary sync [20:07:49] ah, they don't run on mwdebug or deploy host? [20:08:17] I thought the reason the PHP Notice from all requests went to prod (canaries) is because the swagger checks only check for fatals. [20:08:27] so I was gonna find a way to make notices fatal on those requests [20:08:33] but that wouldn't help then. [20:08:55] Krinkle: hello. Thank you very much for the scap/canary/eqiad task :] [20:09:01] yeah, we don't run on deploy host since the sites aren't setup there [20:09:39] Aye, but that means the swagger checks aren't able to catch things earlier. Unless the real traffic is not hitting the issue our swagger checks hit, which is very unlikely. [20:09:41] running swagger checks on mwdebug hosts before the rest of the canaries might be easy to achieve [20:10:20] additional dsh group and an extra step in scap [20:12:47] https://phabricator.wikimedia.org/T121597#4603654 [20:12:48] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597 (10Krinkle) Ping. The above one-liner would've prevented the following from affecting production traffic to canary hosts today. > https://gerrit.wikimedia.org/r/#/c... [20:13:02] thcipriani: Sounds good to me. [20:13:28] * thcipriani makes subtask [20:14:30] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5) - https://phabricator.wikimedia.org/T191921 (10hashar) Great! `\o/` [20:16:26] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:17] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [20:58:41] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release Pipeline (Blubber): Blubber: put node_modules in /srv/app rather than /opt/lib - https://phabricator.wikimedia.org/T204591 (10dduvall) 05Open>03Resolved [21:15:56] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.32.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T191068 (10hashar) 05Open>03Resolved 1.32.0-wmf.22 has been rolled to all wikis and seems all fine. Thank you @tgr and @Anomie to have shown... [21:17:04] 10Beta-Cluster-Infrastructure: Generate beta captchas - https://phabricator.wikimedia.org/T204611 (10Krenair) Looks like the higher you increase --fill the more broken it gets. I tried around --fill=1000 and got a bunch of `* An unknown error occurred in storage backend "global-swift-eqiad".` when copying new ca... [21:27:37] twentyafterfour, nice blog post about task types. Thanks! [21:33:43] 10MediaWiki-Releasing, 10Security: Obtain CVE's for 1.27.5/1.29.3/1.30.1/1.31.1 security releases - https://phabricator.wikimedia.org/T199027 (10Reedy) [21:33:57] 10MediaWiki-Releasing, 10Security: Tag MW 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199026 (10Reedy) [21:34:03] 10MediaWiki-Releasing, 10Security: Update HISTORY in master after 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199025 (10Reedy) [21:34:10] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199024 (10Reedy) [21:35:39] 10MediaWiki-Releasing, 10Security: Obtain CVEs for 1.27.5/1.29.3/1.30.1/1.31.1 security releases - https://phabricator.wikimedia.org/T199027 (10Jdforrester-WMF) [21:35:50] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199021 (10Reedy) [21:49:56] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199021 (10Reedy) [21:49:59] 10MediaWiki-Releasing, 10Security: Update onwiki release notes for 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199024 (10Reedy) 05Open>03Resolved a:03Reedy [21:57:37] 10MediaWiki-Releasing, 10Security: Write and send pre-release Announcements for MediaWiki 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199023 (10Reedy) [22:09:13] (03CR) 10MarcoAurelio: [C: 031] "I'm not sure if this can be merged now so giving +1 instead." [tools/release] - 10https://gerrit.wikimedia.org/r/461733 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:09:55] (03CR) 10Reedy: [C: 04-2] "Unfortunately not just yet. CR-2'ing it so it's not accidentally merged" [tools/release] - 10https://gerrit.wikimedia.org/r/461733 (https://phabricator.wikimedia.org/T106067) (owner: 10Reedy) [22:20:20] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199021 (10Reedy) [22:20:22] 10MediaWiki-Releasing, 10Security: Update HISTORY in master after 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199025 (10Reedy) 05Open>03Resolved a:03Reedy [22:20:32] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.27.5/1.29.3/1.30.1/1.31.1 - https://phabricator.wikimedia.org/T199021 (10Reedy) 05Open>03Resolved DONED [22:41:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Evaluate different strategy for Docker CI instances - https://phabricator.wikimedia.org/T202160 (10dduvall) >>! In T202160#4591940, @dduvall wrote: > I've reconfigured one of the `bigram` instances (`integration... [22:53:03] quiddity: thanks! [22:56:22] 10Phabricator, 10User-MModell: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) @mgchecker: The only way to manually change types is using the batch editor. There is also the possibility to build herald rules which either react to the type or set the type based on other criter... [23:04:58] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597 (10mmodell) I'll take a stab at calling `echo 1 | mwscript eval.php` in scap. [23:13:31] (03CR) 10Reedy: "How do we actually test this then?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/461195 (https://phabricator.wikimedia.org/T204569) (owner: 10Reedy) [23:20:13] !log taking integration-slave-docker-1004/1005 offline for replacement (T202160) [23:20:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:20:21] T202160: Evaluate different strategy for Docker CI instances - https://phabricator.wikimedia.org/T202160 [23:20:39] 10Phabricator, 10User-MModell: Add support for task types - https://phabricator.wikimedia.org/T93499 (10MGChecker) >>! In T93499#4604434, @mmodell wrote: > @mgchecker: The only way to manually change types is using the batch editor. There is also the possibility to build herald rules which either react to the... [23:21:40] 10Phabricator, 10User-MModell: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) @mgchecker: no I don't think so. Changing task types shouldn't be the norm and some changes would break things so it really shouldn't be on the normal task edit form. [23:23:52] !log launching integration-slave-docker-1035/1036 bigram instances [23:23:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:27:23] PROBLEM - Host integration-slave-docker-1006 is DOWN: CRITICAL - Host Unreachable (10.68.17.244) [23:29:37] PROBLEM - Host integration-slave-docker-1005 is DOWN: CRITICAL - Host Unreachable (10.68.21.12) [23:43:00] 10Phabricator, 10User-MModell: Add support for task types - https://phabricator.wikimedia.org/T93499 (10MGChecker) But if we want to actually replace the mentioned projects with task types, I think such a functionality would really be needed, as after all these task types are especially useful if they are appl... [23:43:05] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597 (10mmodell) [23:48:07] !log adding new integration-slave-docker-1035/1036 jenkins nodes, each with 7 executors [23:48:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL