[00:12:30] 10Beta-Cluster-Infrastructure: Captchas sent with wrong mime type on beta - https://phabricator.wikimedia.org/T164047#4021374 (10Krenair) The answer probably lies in some python-swiftclient or python-requests version used by the Trusty -ms-fe01 host on which I originally did the migration (-ms-fe02 came along in... [01:05:27] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4021427 (10Krenair) The instructions for handling such a change, if a developer decides to accept this request... [01:09:33] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4021440 (10Krenair) @bawolff, is that site approved to sit under wikimedia.org, seeing as it has wgRawHtml? [01:24:27] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4021466 (10Varnent) The new scope for that site will result in far fewer pages (essentially just documentation... [01:30:34] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4021476 (10Krenair) Okay but the existing site presumably has to continue to live somewhere, and the special c... [01:35:45] 10Differential, 10Gerrit-Migration: Consider documenting the Arcanist installer for Windows and Mac that Paladox created - https://phabricator.wikimedia.org/T131939#4021492 (10TerraCodes) 05Open>03Resolved a:03TerraCodes I think this is resolved since the installer looks to be documented on mw.org. [01:36:17] 10Differential, 10Gerrit-Migration: Consider documenting the Arcanist installer for Windows and Mac that Paladox created - https://phabricator.wikimedia.org/T131939#4021495 (10TerraCodes) a:05TerraCodes>03None (undoing auto claim) [01:43:14] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4021506 (10Krenair) (Some of) the stuff I wondered across with a quick grep that doesn't look to be covered by... [01:49:20] 10Beta-Cluster-Infrastructure: Captchas sent with wrong mime type on beta - https://phabricator.wikimedia.org/T164047#4021518 (10Krenair) Couldn't quite just use https://wikitech.wikimedia.org/wiki/Generating_CAPTCHAs because it expects prod (antispam) secrets to be in place (the ones in labs/private.git are pre... [01:52:26] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<11.11%) [02:13:49] 10Beta-Cluster-Infrastructure: Captchas sent with wrong mime type on beta - https://phabricator.wikimedia.org/T164047#4021553 (10Krenair) 05Open>03Resolved a:03Krenair Well that did the trick ``` krenair@deployment-tin:/srv/mediawiki-staging/php-master/extensions/ConfirmEdit$ mwscript extensions/ConfirmEdi... [02:26:45] paladox: yeah, are we still using it? I had thought we moved most of it over to just straight chromium/firefox by this point [02:52:48] legoktm: I’m not sure if we are using it, just passing on that thing from upstream in case :) [03:19:42] It doesn't look like it's used by anyone? There's still some remnants in int/config, though [03:31:19] https://codesearch.wmflabs.org/search/?q=phantomjs&i=nope&files=&repos= maybe browser tests? [03:32:16] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4021611 (10Bawolff) >>! In T188776#4021440, @Krenair wrote: > @bawolff, is that site approved to sit under wik... [03:43:29] 10Gerrit, 10Upstream: Polygerrit search dropdown does not list all projects - https://phabricator.wikimedia.org/T188842#4021632 (10Tgr) Thanks for checking! [03:45:35] 10Release-Engineering-Team, 10DNS, 10Operations, 10Traffic, and 2 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776#4021634 (10Varnent) > That said, I would certainly prefer it wasnt a raw html wiki. At first glance it sounds... [04:50:29] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:20:30] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [05:59:55] no_justification: Hm.. was php-master on beta/deployment-tin recreated or something? [06:00:02] It seems StartProfiler.php no longer exists [06:00:27] I had to manually create it there sometime last year because that dir isn't scap or puppet-ified afaik [06:01:03] Shit. I have no idea [06:02:53] !log Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead. [06:02:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:03:07] I was wondering why profiling didnt' work anymore [06:03:23] spent like 2 hours figuring out what I could've misspelled or did wrong in my recente deployment a few days ago [06:03:35] Then to realise the file isn't loaded on Beta :D [06:03:42] Project beta-scap-eqiad build #198112: 04FAILURE in 0.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198112/ [06:04:25] ^ simple lock failure because I'm already doing a scap sync-file from deployment-tin [06:05:06] OK Profiling works again now [06:05:22] And it seems my patch didn't break it :P [06:05:35] no_justification: Hm.. so how *is* php-master created anyway? [06:05:49] Is it a manual thing from time to time? Do you recall ever re-creating it? [06:06:30] I assume that whatever does that, would also create LocalSettings.php, which does exist (beta'd be down otherwise) [06:06:36] s/whatever/whoever/ [06:06:42] Well technically scap prep and friends handle it [06:06:45] Would be good to document (where?) to also ensure StartProfiler [06:06:48] But it's so rare [06:06:53] It lives forever [06:06:57] Righ [06:06:58] t [06:07:01] Is there code for it in scap? [06:07:12] Or... does the regular branch code work for that? [06:08:34] Wmf config repo [06:09:05] Scap/plugins/* [06:09:11] Yippee, build fixed! [06:09:12] Project beta-scap-eqiad build #198113: 09FIXED in 4 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198113/ [06:15:04] no_justification: https://github.com/wikimedia/operations-mediawiki-config/blob/5ac46adf6f4ed7fc0ff693d7c90426d313e9ad5a/scap/plugins/prep.py#L49 [06:15:05] Interesting [06:15:13] According to that, it would create StartProfiler the same way in prod [06:15:18] which... I'm pretty sure doesn't work [06:15:54] or at least, it shouldn't. [06:16:11] I guess we can semi-assume that ==master means == beta [06:16:18] and maybe fragment the StartProfile creation there [06:16:31] to write an include for StartProfiler-labs instead [06:19:57] Go for it.... [08:03:53] (03CR) 10Umherirrender: [C: 032] Update squizlabs/php_codesniffer to 3.2.3 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416214 (owner: 10Legoktm) [08:04:59] (03Merged) 10jenkins-bot: Update squizlabs/php_codesniffer to 3.2.3 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416214 (owner: 10Legoktm) [08:05:09] (03CR) 10jenkins-bot: Update squizlabs/php_codesniffer to 3.2.3 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416214 (owner: 10Legoktm) [08:29:31] (03PS1) 10Umherirrender: Validate @license against SPDX [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 [08:36:07] (03CR) 10jerkins-bot: [V: 04-1] Validate @license against SPDX [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 (owner: 10Umherirrender) [08:37:49] (03PS2) 10Umherirrender: Validate @license against SPDX [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 [08:38:12] (03CR) 10Umherirrender: "Patch Set 2: Removed unused use" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/416337 (owner: 10Umherirrender) [08:50:09] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:50:57] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:25:32] 10Phabricator (Upstream), 10Upstream: Create the user-physikerwelt personal project - https://phabricator.wikimedia.org/T188855#4021861 (10Physikerwelt) p:05Triage>03Lowest [10:27:09] 10Project-Admins: Create the user-physikerwelt personal project - https://phabricator.wikimedia.org/T188855#4021874 (10Jayprakash12345) [12:54:14] 10Beta-Cluster-Infrastructure, 10Privacy, 10User-MarcoAurelio: Disable the collection of private information on abusefilter log for Beta Cluster wikis - https://phabricator.wikimedia.org/T188862#4022014 (10MarcoAurelio) [13:03:09] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Privacy, 10User-MarcoAurelio: Disable the collection of private information on abusefilter log for Beta Cluster wikis - https://phabricator.wikimedia.org/T188862#4022014 (10MarcoAurelio) p:05Triage>03Normal [15:06:57] 10Gerrit, 10Upstream: Polygerrit search dropdown does not list all projects - https://phabricator.wikimedia.org/T188842#4022120 (10Paladox) For it searching too quickly, i think that is known, but has not been fixed yet. For the description being in the search result, i doin't think that has been noted, would... [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:38:20] (03PS1) 10Paladox: Grant Edit hashtag right to everyone. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 [18:39:21] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:47:36] 10Gerrit, 10Upstream: Allow rebase in gerrit following renames - https://phabricator.wikimedia.org/T188192#4022320 (10Paladox) Filled upstream at https://bugs.chromium.org/p/gerrit/issues/detail?id=8455 [18:47:49] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) [18:51:08] Project beta-scap-eqiad build #198188: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198188/ [18:53:24] 18:51:03 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-tin.deployment-prep.eqiad.wmflabs::common/wikiversions*.{json,php} /srv/mediawiki on deployment-mediawiki04.deployment-prep.eqiad.wmflabs returned [11]: rsync: write failed on "/srv/mediawiki/wikiversions-labs.php": No space left on device (28) [18:53:24] 18:51:03 rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1] [18:53:32] Reedy thcipriani ^^ [19:00:32] Yippee, build fixed! [19:00:33] Project beta-scap-eqiad build #198189: 09FIXED in 8 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/198189/ [19:00:38] !log cleared /tmp, apt-cache deployment-mediawiki04 [19:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:06:31] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [19:28:19] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Privacy, 10User-MarcoAurelio: Disable the collection of private information on abusefilter log for Beta Cluster wikis - https://phabricator.wikimedia.org/T188862#4022014 (10Krenair) I don't think this will actually prevent people without NDAs from access... [19:30:12] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Privacy, 10User-MarcoAurelio: Disable the collection of private information on abusefilter log for Beta Cluster wikis - https://phabricator.wikimedia.org/T188862#4022369 (10MarcoAurelio) afaics that variable will prevent abusefilter from collecting that... [19:31:59] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Privacy, 10User-MarcoAurelio: Disable the collection of private information on abusefilter log for Beta Cluster wikis - https://phabricator.wikimedia.org/T188862#4022372 (10Krenair) sure but they could turn it back on, or dig up IPs etc. some other way [19:52:55] 10Beta-Cluster-Infrastructure: Requesting PCR on enwiki beta - https://phabricator.wikimedia.org/T188873#4022412 (10Sau226) [19:53:31] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [19:57:31] 10Beta-Cluster-Infrastructure: Suggestion: Give stewards userrights-interwiki on deploymentwiki and remove the global userrights, userrights-interwiki - https://phabricator.wikimedia.org/T188874#4022425 (10Sau226) [20:02:55] 10Beta-Cluster-Infrastructure: Requesting PCR on enwiki beta - https://phabricator.wikimedia.org/T188873#4022436 (10Sau226) Also while you're on it could you at please unprotect my global user page on there and set it to autoconfirmed editing? [20:39:03] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:47:11] 10Beta-Cluster-Infrastructure: Requesting PCR on enwiki beta - https://phabricator.wikimedia.org/T188873#4022443 (10Aklapper) What is "[[ https://en.wikipedia.org/wiki/PCR_(disambiguation) | PCR ]]"? [20:50:17] 10Gerrit, 10Upstream: Polygerrit search dropdown does not list all projects - https://phabricator.wikimedia.org/T188842#4022445 (10Paladox) I've created https://gerrit-review.googlesource.com/c/gerrit/+/163990 to fix the fasteness of when you type in project: [22:22:25] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<55.56%) [22:25:14] legoktm ^^