[04:58:40] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [06:58:40] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:20:38] 10Phabricator: Calendar monthly view makes event names unreadable - https://phabricator.wikimedia.org/T187019#3962075 (10Smalyshev) [07:29:49] (03CR) 10Legoktm: Add sniff to find tests without @covers tags (034 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/409444 (https://phabricator.wikimedia.org/T179094) (owner: 10Legoktm) [07:30:07] (03PS2) 10Legoktm: Add sniff to find tests without @covers tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/409444 (https://phabricator.wikimedia.org/T179094) [07:46:53] (03PS1) 10Chad: Adding Sentry to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/409749 [08:33:30] Project mwext-phpunit-coverage-publish build #815: 04FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/815/ [08:33:59] Yippee, build fixed! [08:34:00] Project mwext-phpunit-coverage-publish build #816: 09FIXED in 29 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/816/ [08:57:22] (03PS1) 10Hashar: Migrate operations-puppet-doc to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409814 [08:57:36] (03CR) 10Hashar: [C: 032] Migrate operations-puppet-doc to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409814 (owner: 10Hashar) [09:00:02] (03Merged) 10jenkins-bot: Migrate operations-puppet-doc to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409814 (owner: 10Hashar) [09:15:11] zeljkof: good morning! Is there anything left still using the mediawiki_selenium gem ? :] [09:16:35] hashar: half of the repos are still using it [09:16:58] zeljkof: guess so I have to migrate the mediawiki-selenium-integration job which uses a locally setup mediawiki :] [09:22:59] hashar: well, we did deprecate Ruby stack [09:23:44] I would say this is the right time to decide we are not putting any more effort into it [09:24:31] If people want to run Selenium tests, they have to move to node [09:33:19] zeljkof: got it :] I might use that job as a first candidate to migrate the mediawiki tests to docker [09:44:32] (03CR) 10Thiemo Kreuz (WMDE): Add sniff to find tests without @covers tags (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/409444 (https://phabricator.wikimedia.org/T179094) (owner: 10Legoktm) [10:01:48] 10Continuous-Integration-Infrastructure, 10VisualEditor, 10Wikidata, 10Jenkins, and 2 others: ResourcesTest::testFileExistence fails for ULS files in VisualEditor - https://phabricator.wikimedia.org/T186903#3962557 (10Lucas_Werkmeister_WMDE) Okay, thanks for fixing :) [10:04:18] 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10Zuul: Upload new zuul and jenkins-debian-glue packages to apt.wikimedia.org - https://phabricator.wikimedia.org/T186786#3962573 (10fgiunchedi) p:05Triage>03Normal [10:30:05] (03PS1) 10Hashar: Migrate tox-py27-coverage-publish to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409836 [10:30:19] 10Phabricator, 10WMSE-Bug-Reporting-and-Translation-2018: Trying to import workboard for some projects give error 400 - https://phabricator.wikimedia.org/T186189#3936461 (10Lokal_Profil) The behaviour has changed somewhat. Now I get the following unhandled exception instead `Array for %Ls conversion is empty.... [10:42:54] (03CR) 10Hashar: [C: 032] "https://doc.wikimedia.org/cover/labs-tools-heritage/ \o/" [integration/config] - 10https://gerrit.wikimedia.org/r/409836 (owner: 10Hashar) [10:44:21] (03Merged) 10jenkins-bot: Migrate tox-py27-coverage-publish to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409836 (owner: 10Hashar) [10:51:04] 10Beta-Cluster-Infrastructure: Spam/Spambot registration on deployment.beta is out of hand - https://phabricator.wikimedia.org/T187046#3962785 (10MarcoAurelio) [10:55:12] Reedy: mwscript cleanupSpam.php seems recently broken for some reason [10:55:20] Error? [10:55:27] no domains found [10:55:34] and they are [10:55:45] maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.loginist.com --delete [10:55:45] Found 0 articles containing *.loginist.com [10:55:58] https://deployment.wikimedia.beta.wmflabs.org/wiki/How_To_Get_A_Pnc_Pathfinder_Login [10:56:10] it's the https:// thingy? [10:57:37] Is it? [10:57:48] wgServer and wgCanonicalServer are both HTTPS [10:58:47] I mean the spam domain [10:59:06] however using the full spam domain https://spametcetc... doesn't work either [10:59:28] https:// loginist.com [11:00:25] maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki https://loginist.com/ --delete [11:00:25] Found 0 articles containing https://loginist.com/ [11:02:19] apparently it queries el_index from externallinks [11:06:21] el_index have the domain https://phabricator.wikimedia.org/P6680 [11:09:30] wikiadmin@deployment-db04[deploymentwiki]> select count(*) from externallinks where el_index like '%loginist%'; finds it [11:09:32] hmm [11:09:41] error in the query script? [11:12:49] The code hasn't changed recently AFAIK [11:13:12] https://github.com/wikimedia/mediawiki/commits/master/maintenance/cleanupSpam.php [11:13:42] does %.loginist.com work? [11:14:55] let me check [11:15:40] Found 0 articles containing %.loginist.com [11:15:51] it supports a '*' only [11:15:59] and with '*' it doesn't work either [11:16:37] https://tools.wmflabs.org/sal/releng?p=0&q=cleanupSpam.php&d= [11:16:43] that shows I could run it before [11:17:07] The code hasn't changed [11:18:31] which is the exact sql query the script makes? [11:18:44] I think it might be domains being spammed on https rather than on http [11:18:48] but I could be wrong [11:19:53] makeLikeArray only uses http [11:20:03] testing on https://deployment.wikimedia.beta.wmflabs.org/wiki/Spam_test -- switched to http [11:20:05] So I guess that's the issue [11:20:16] moment [11:20:31] yep, with http the script works [11:20:42] https://deployment.wikimedia.beta.wmflabs.org/w/index.php?title=Spam_test&action=history [11:21:14] I'll make a patch [11:22:12] https://p.defau.lt/?RVm8vJzylFbDO9y2pbn9DQ [11:22:59] T186795 [11:23:00] T186795: cleanupSpam.php fails to work if domain is not http:// - https://phabricator.wikimedia.org/T186795 [11:23:33] should also detect things like ftp:// etc [11:23:40] but for not it's a good improvement [11:24:43] bah [11:24:44] if ( count( $params ) > 0 && is_array( $params[0] ) ) { [11:24:44] $params = $params[0]; [11:24:44] } [11:24:47] Can't do it that way... [11:25:59] 10Gerrit, 10Wikibase-DataModel, 10Wikidata, 10Wikidata-Sprint-2016-03-01, and 2 others: [Task] Move WikibaseDataModel to Gerrit - https://phabricator.wikimedia.org/T126597#3962915 (10WMDE-leszek) Thanks for ping @Addshore. I have no strong opinion here. Some rambling below. Description mentions there are... [11:31:08] Hauskatze: Patch [11:31:26] Reedy: *hug* [11:31:28] thanks :) [11:31:30] Though [11:31:33] If we're gonna do that... [11:32:24] PS2 [11:37:12] Probably gonna be slower... [11:37:43] well, if it works, I don't care much if it's few seconds slower :) [11:38:10] Helps prevent that bitcoin wallet spam [11:38:10] xD [11:38:21] Project mwext-phpunit-coverage-publish build #823: 04FAILURE in 36 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/823/ [11:43:32] bah [11:46:23] (03PS1) 10MarcoAurelio: zuul: archive the Wiki Object Model extension [integration/config] - 10https://gerrit.wikimedia.org/r/409851 (https://phabricator.wikimedia.org/T186516) [11:51:50] Yippee, build fixed! [11:51:51] Project mwext-phpunit-coverage-publish build #824: 09FIXED in 56 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/824/ [11:54:31] 10Gerrit, 10Wikibase-DataModel, 10Wikidata, 10Wikidata-Sprint-2016-03-01, and 2 others: [Task] Move WikibaseDataModel to Gerrit - https://phabricator.wikimedia.org/T126597#3962955 (10Addshore) As a 'third party user' I have no real preference for the location, as long as it is installable via composer. [11:58:20] thanks Reedy for your help [12:00:09] 10Phabricator: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3962963 (10Elitre) [12:06:28] 10Phabricator: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3962994 (10Reedy) [12:07:10] 10Phabricator: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3962963 (10Reedy) The two tasks are procurement tasks (that are resolved), so probably aren't tasks that could be opened up to the public [12:17:00] Project beta-scap-eqiad build #195118: 04FAILURE in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195118/ [12:18:49] (03PS1) 10Hashar: Migrate mw-tools-codesniffer-mwcore-testrun to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409863 [12:19:03] 10Phabricator: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3962963 (10MarcoAurelio) Shouldn't they appear as "Restricted Task" as in other trees I saw? [12:23:13] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1343 bytes in 0.008 second response time [12:23:45] (03CR) 10Hashar: [C: 032] Migrate mw-tools-codesniffer-mwcore-testrun to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409863 (owner: 10Hashar) [12:25:06] (03Merged) 10jenkins-bot: Migrate mw-tools-codesniffer-mwcore-testrun to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409863 (owner: 10Hashar) [12:26:59] Project beta-scap-eqiad build #195119: 04STILL FAILING in 3 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195119/ [12:28:20] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 47124 bytes in 8.574 second response time [12:37:07] Project beta-scap-eqiad build #195120: 04STILL FAILING in 3 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195120/ [12:40:57] Reedy: so it seems the abusefilter script is already running? T187053 ? [12:40:58] T187053: Setup puppet cron to delete old data daily - https://phabricator.wikimedia.org/T187053 [12:41:05] weird [12:41:35] I can have a look in a bit... [12:41:43] cannot really test on production, maybe someone with access can query the abuse_filter_log table for an old entry and see if there's any afl_ip data there [12:41:47] ok [12:41:54] lunch time for me [12:42:08] otoh, it's 'foreachwiki', don't that work on Beta too? [12:45:45] 10Beta-Cluster-Infrastructure, 10Operations: Remove video scaler instances from deployment-prep - https://phabricator.wikimedia.org/T187063#3963166 (10MoritzMuehlenhoff) [12:47:09] Project beta-scap-eqiad build #195121: 04STILL FAILING in 3 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195121/ [12:47:52] It should [12:50:35] FAILING [12:50:48] STILL [12:50:50] Reedy: fix it [12:50:57] * Reedy slaps addshore [12:51:04] * addshore chunks Reedy into the Reeds [12:51:19] http://canoetrips.co.uk/Idle/reeds.JPG [12:51:30] Now your SuperReeedy! [12:51:55] 12:46:49 rsync: failed to set times on "/srv/mediawiki/.": Operation not permitted (1) [12:51:56] 12:46:49 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [generator=3.1.1] [12:51:59] Permissions got mangled? [12:52:26] Or just deployment-jobrunner02 fucked up? [12:53:42] reedy@deployment-jobrunner02:~$ scap pull [12:53:43] 12:53:30 Copying to deployment-jobrunner02.deployment-prep.eqiad.wmflabs from deployment-tin.deployment-prep.eqiad.wmflabs [12:53:43] 12:53:30 Started rsync common [12:53:43] 12:53:33 Finished rsync common (duration: 00m 02s) [12:53:59] WFM? [12:54:34] run that as jenkins? [12:54:49] sudo -u jenkins-deploy scap pull (or whatever) [12:55:16] i think thats the right user >.> [12:55:35] root@deployment-jobrunner02:/home/reedy# sudo -u jenkins-deploy scap pull [12:55:35] Creating directory '/mnt/home/jenkins-deploy'. [12:55:35] 12:55:26 Copying to deployment-jobrunner02.deployment-prep.eqiad.wmflabs from deployment-tin.deployment-prep.eqiad.wmflabs [12:55:35] 12:55:26 Started rsync common [12:55:35] 12:55:29 Finished rsync common (duration: 00m 03s) [12:55:52] mwdeploy is the right user [12:55:54] i think [12:56:11] thats the user that owns the files in /srv/mediawiki [12:57:01] Yippee, build fixed! [12:57:02] Project beta-scap-eqiad build #195122: 09FIXED in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195122/ [12:57:19] Reedy: everything looks fine to me :D [12:59:34] fucking computers [13:14:47] (03PS1) 10Hashar: Migrate integration-config-qa to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409879 [13:15:16] (03PS2) 10Hashar: Migrate integration-config-qa to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409879 [13:15:43] (03CR) 10Hashar: [C: 032] Migrate integration-config-qa to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409879 (owner: 10Hashar) [13:17:41] (03Merged) 10jenkins-bot: Migrate integration-config-qa to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409879 (owner: 10Hashar) [13:30:24] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Puppet, 10User-Addshore: mediawiki::maintenance::wikidata should not run crons for testwikidatawiki when used on labs / a testwikidatawiki doesnt exist - https://phabricator.wikimedia.org/T173357#3963295 (10Addshore) [13:44:24] 10Phabricator (Upstream), 10Upstream: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3963351 (10Aklapper) p:05Triage>03Low Looks like https://secure.phabricator.com/T4345 striking again [13:53:06] 10Phabricator (Upstream), 10Upstream: Calendar monthly view makes event names unreadable - https://phabricator.wikimedia.org/T187019#3963377 (10Aklapper) p:05Triage>03Low > you can see that event names have only two letters displayed. Can only confirm that on 1024px screen width. Workaround: Zoom out. See... [14:00:34] (03PS1) 10Hashar: docker: java8 + maven image on Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/409881 [14:04:36] 10Phabricator (Upstream), 10Upstream: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3963425 (10Elitre) >>! In T187051#3963351, @Aklapper wrote: > Looks like https://secure.phabricator.com/T4345 striking again I don't wan... [14:29:53] 10Scap: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T187076#3963501 (10fgiunchedi) [14:31:37] 10Scap, 10Operations: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T187076#3963515 (10fgiunchedi) p:05Triage>03Normal I'll hold back fixing naos too to leave time #release-engineering-team to inspect the situation. [14:31:51] 10Scap, 10Operations: Deploy error: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T187076#3963518 (10fgiunchedi) [15:25:45] 10Diffusion, 10MediaWiki-extensions-CentralNotice, 10Repository-Admins: Diffusion link for CentralNotice on Special:Version doesn't work - https://phabricator.wikimedia.org/T187077#3963715 (10Paladox) [15:26:49] Hello peopl1 [15:26:53] *people! [15:27:45] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:28:09] is twentyafterfour around today? [15:33:42] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:50:14] 10Phabricator (Upstream), 10Upstream: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3963847 (10Aklapper) >>! In T187051#3963425, @Elitre wrote: > I don't want to be able to see what's in those tasks: IMO it should say "R... [15:57:37] (03CR) 10Hashar: [C: 032] docker: java8 + maven image on Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/409881 (owner: 10Hashar) [15:58:45] (03Merged) 10jenkins-bot: docker: java8 + maven image on Stretch [integration/config] - 10https://gerrit.wikimedia.org/r/409881 (owner: 10Hashar) [15:59:11] Project mwext-phpunit-coverage-publish build #829: 04FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/829/ [15:59:15] !log Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/ [15:59:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:04:11] Yippee, build fixed! [16:04:12] Project mwext-phpunit-coverage-publish build #830: 09FIXED in 38 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/830/ [16:05:27] 10Phabricator (Upstream), 10Upstream: "Unknown Object (Task)" in dependency tree doesn't allow regular users to figure out status - https://phabricator.wikimedia.org/T187051#3963928 (10Elitre) If it matters that security stuff isn't distinguishable from, say, legal stuff, I don't have a problem with that and y... [16:08:36] 10Scap, 10Operations: Deploy error: insufficient permission for adding an object to repository database .git/objects - https://phabricator.wikimedia.org/T187076#3963501 (10demon) This usually happens for one of two reasons # A root user has come along and stolen ownership to root. This should't happen often in... [16:20:58] (03PS1) 10Hashar: docker: fix java8 entrypoint [integration/config] - 10https://gerrit.wikimedia.org/r/409934 [16:27:26] no_justification i wonder does this look ok to you https://github.com/GerritCodeReview/plugins_its-phabricator/commit/a86babaae85d5c698beb9e520bbc03652b93ca24#diff-25d902c24283ab8cfbac54dfa101ad31 [16:27:27] ? [16:45:21] (03CR) 10Chad: "I don't have the time to work on this, and the priority is so incredibly low. No point in cluttering my dashboard." [integration/config] - 10https://gerrit.wikimedia.org/r/404753 (owner: 10Chad) [16:50:31] 10Gerrit, 10Wikibase-DataModel, 10Wikidata, 10Wikidata-Sprint-2016-03-01, and 2 others: [Task] Move WikibaseDataModel to Gerrit - https://phabricator.wikimedia.org/T126597#3964211 (10demon) >>! In T126597#3962915, @WMDE-leszek wrote: > I understand it might make sense from WMF point of view as "all things... [16:54:21] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T183959#3964235 (10demon) [16:54:24] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T183959#3868515 (10demon) [17:03:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T183959#3964269 (10Jdforrester-WMF) [17:09:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Provision blubber on CI labs instances - https://phabricator.wikimedia.org/T187095#3964281 (10dduvall) [17:10:16] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3964292 (10dduvall) [17:10:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Provision blubber on CI labs instances - https://phabricator.wikimedia.org/T187095#3964291 (10dduvall) [17:11:06] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Add blubber to docker integration agents - https://phabricator.wikimedia.org/T186548#3964296 (10dduvall) a:03dduvall [17:11:23] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Add blubber to docker integration agents - https://phabricator.wikimedia.org/T186548#3947012 (10dduvall) p:05Triage>03Normal [17:13:40] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3964316 (10dduvall) [17:13:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Provision blubber on CI labs instances - https://phabricator.wikimedia.org/T187095#3964312 (10dduvall) 05Open>03declined Whoops. Duplicate of {T186548} [17:13:57] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#3845669 (10dduvall) [17:14:00] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Add blubber to docker integration agents - https://phabricator.wikimedia.org/T186548#3947012 (10dduvall) [17:18:50] (03PS1) 10Hashar: jjb: Docker run_args no more require a space [integration/config] - 10https://gerrit.wikimedia.org/r/409955 [17:18:52] (03PS1) 10Hashar: Migrate mwdumper-maven to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/409956 [17:19:37] (03CR) 10ArielGlenn: "No objections, just wondering what this will mean in practice." [integration/config] - 10https://gerrit.wikimedia.org/r/409956 (owner: 10Hashar) [17:28:28] 10Scap: Add DEPLOY_DIR env var to scap command checks - https://phabricator.wikimedia.org/T154612#3964392 (10mmodell) [17:29:28] 10Phabricator, 10Developer-Relations: Find best way to allow CoC committee to temporarily disable Phab user accounts - https://phabricator.wikimedia.org/T183116#3964394 (10Aklapper) a:03Aklapper [17:32:54] 10Continuous-Integration-Infrastructure, 10Release Pipeline (Blubber): jenkins-slave@contint1001 not a member of docker group (CI tests for mathoid broken) - https://phabricator.wikimedia.org/T186790#3964406 (10akosiaris) A puppet run on contint1001 currently first adds `jenkins-slave` to the docker group and... [17:35:48] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3964413 (10mmodell) [17:35:51] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3964414 (10mmodell) [17:39:34] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3949613 (10mmodell) I think this is probably the same root cause as T182832: that one is an unso... [17:40:59] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3964442 (10mmodell) I've been away for 2 weeks due to our team offsite, travel and vacation time... [17:42:39] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3964444 (10elukey) >>! In T186620#3964442, @mmodell wrote: > I've been away for 2 weeks due to o... [17:46:38] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3964452 (10Paladox) I think upstream test with php 7.1 now (even though php 5.2+ is supported) s... [17:54:51] (03PS4) 10Niedzielski: Promote chromium-render from experimental [integration/config] - 10https://gerrit.wikimedia.org/r/409115 (https://phabricator.wikimedia.org/T179552) [18:27:10] Project beta-scap-eqiad build #195157: 04FAILURE in 3 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195157/ [18:31:02] When did the train actually go out? [18:31:04] Was it on Friday? [18:31:21] no_justification should know? :-) [18:37:25] Project beta-scap-eqiad build #195158: 04STILL FAILING in 3 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195158/ [18:37:39] Deskana: it finally made it everywhere on Friday, yes. http://tools.wmflabs.org/sal/log/AWF71dyokkJ8OkTwh_SU [18:38:05] Thanks Greg! [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:50] Deskana: What greg said :) [18:40:37] no_justification: did you got my PM (hi btw) from the other day? [18:43:58] !log Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - T186870 [18:44:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:44:05] T186870: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870 [18:44:21] !log maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged) [18:44:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:45:24] !log maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - T186870) [18:45:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:45:35] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Get MediaWIki running in Docker with Blubber - https://phabricator.wikimedia.org/T187105#3964707 (10thcipriani) [18:46:15] !log maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - T186870) [18:46:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:47:03] Project beta-scap-eqiad build #195159: 04STILL FAILING in 3 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195159/ [18:48:14] !log Restoring first missing log entries on Beta refs. T54919 [18:48:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:48:27] T54919: Data about oldest revisions of edit filters from (some?) WMF wikis is not available in the logs - https://phabricator.wikimedia.org/T54919 [18:48:41] Hauskatze: Yeah, I did. There's gotta be some other permission overriding things? [18:49:15] no_justification: most likely as I don't get why I have 'submit' in All Projects but can't submit [18:53:38] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-MarcoAurelio: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870#3964737 (10MarcoAurelio) P6684 - I added some missing rows. [18:55:06] !log Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for T186870 [18:55:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:55:12] T186870: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870 [18:57:09] Project beta-scap-eqiad build #195160: 04STILL FAILING in 3 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195160/ [18:58:35] Hauskatze: I wonder if you can submit on a change on All-Projects itself? [18:59:00] no_justification: haven't tested, want me to? [18:59:05] Worth a shot [18:59:09] Would answer our question, methinks [18:59:24] okay, let me paste the output of the foreachwikiindblist script I've just ran [19:00:21] Also: I wonder if you can actually do submit on refs/meta/config or if it's a weird reference that requires project ownership too? [19:00:43] Wouldn't be the oddest thing ever.... [19:01:12] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-MarcoAurelio: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870#3964787 (10MarcoAurelio) Done: {P6685} However we should either disabling `afl_ip` logging on Beta Cluster, or restrict who can access that i... [19:01:29] no_justification: I can't submit on refs/meta/config, now doing the test for all projects [19:02:51] (03Draft1) 10MarcoAurelio: test [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/410002 [19:02:53] (03PS2) 10MarcoAurelio: test [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/410002 [19:03:11] now c2+v2 and see if I see the 'submit' button [19:03:23] (03CR) 10MarcoAurelio: [V: 032 C: 032] "testing with Chad monitoring" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/410002 (owner: 10MarcoAurelio) [19:03:34] and the submit button does not appear [19:05:16] no_justification: ^ [19:05:45] Bleh [19:06:32] maybe the refs/meta/* takes precedence? [19:07:13] Project beta-scap-eqiad build #195161: 04STILL FAILING in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195161/ [19:09:56] (03PS1) 10Legoktm: ci-src-setup: Install extension's composer dependencies too [integration/config] - 10https://gerrit.wikimedia.org/r/410005 (https://phabricator.wikimedia.org/T186315) [19:10:45] Project beta-scap-eqiad build #195162: 04STILL FAILING in 3 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195162/ [19:15:21] Project beta-scap-eqiad build #195163: 04STILL FAILING in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195163/ [19:16:49] 10Gerrit, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Clean up mediawiki/extensions/Wikibase ACL - https://phabricator.wikimedia.org/T183081#3964834 (10Legoktm) 05Open>03Resolved a:03Legoktm OK, I've reset it to the standard ACL. Please let me know / re-open if you run into any issues. [19:33:53] Project beta-scap-eqiad build #195164: 04STILL FAILING in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195164/ [19:40:21] Project beta-scap-eqiad build #195165: 04STILL FAILING in 5 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195165/ [19:42:15] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3964968 (10mmodell) >>! In T182832#3945200, @elukey wro... [19:45:55] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3964973 (10mmodell) It seems that something has changed... [19:47:25] Project beta-scap-eqiad build #195166: 04STILL FAILING in 3 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195166/ [19:53:35] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-MarcoAurelio: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870#3965005 (10Huji) @MarcoAurelio what was the total execution time? Total row count is 128K [19:55:00] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965012 (10mmodell) >>! In T182832#3948659, @Paladox wr... [19:57:24] Project beta-scap-eqiad build #195167: 04STILL FAILING in 3 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195167/ [19:57:33] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3965021 (10mmodell) The php stack trace points to phabricator setup checks which should only hap... [20:00:10] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10User-Elukey: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array" - https://phabricator.wikimedia.org/T186620#3965024 (10mmodell) I guess it must be related to the parent task - it's in ExecFuture which is... [20:07:23] Project beta-scap-eqiad build #195168: 04STILL FAILING in 3 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195168/ [20:09:51] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965043 (10mmodell) >>! In T182832#3947150, @elukey wro... [20:10:28] 10Gerrit: Outgoing and recently closed reviews that have new comments should be marked as bold just as the incoming reviews - https://phabricator.wikimedia.org/T131968#3965044 (10Paladox) Upstream are redesigning the dashboard in polygerrit. https://bugs.chromium.org/p/gerrit/issues/detail?id=8362 Would you li... [20:11:11] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965045 (10Paladox) @mmodell option one would at least... [20:11:53] 10Gerrit: Outgoing and recently closed reviews that have new comments should be marked as bold just as the incoming reviews - https://phabricator.wikimedia.org/T131968#3965046 (10bmansurov) @Paladox yes, that would be awesome. Thanks! [20:12:33] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965049 (10mmodell) @Paladox: I don't think stretch upg... [20:12:40] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T183959#3965052 (10demon) [20:13:24] 10Gerrit: Outgoing and recently closed reviews that have new comments should be marked as bold just as the incoming reviews - https://phabricator.wikimedia.org/T131968#3965059 (10Paladox) Done. [20:14:13] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965060 (10Paladox) @mmodell though the old php 5.x is... [20:14:53] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965061 (10mmodell) We don't need php 5.x for phabricat... [20:16:23] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965065 (10Paladox) Yep. We should upgrade to php 7.1 a... [20:17:13] Project beta-scap-eqiad build #195169: 04STILL FAILING in 3 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195169/ [20:17:20] (03CR) 10Umherirrender: [C: 031] Add sniff to find tests without @covers tags (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/409444 (https://phabricator.wikimedia.org/T179094) (owner: 10Legoktm) [20:26:46] Yippee, build fixed! [20:26:46] Project selenium-Wikibase-chrome ยป chrome,beta,Linux,DebianJessie && contintLabsSlave build #104: 09FIXED in 39 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/104/ [20:27:16] Project beta-scap-eqiad build #195170: 04STILL FAILING in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195170/ [20:37:11] Yippee, build fixed! [20:37:12] Project beta-scap-eqiad build #195171: 09FIXED in 3 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/195171/ [20:37:20] !log removed weird leftover rsync .~tmp~ directory on deployment-jobrunner02 to fix beta-scap-eqiad [20:38:59] PROBLEM - Puppet errors on deployment-videoscaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:39:01] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:44:35] (03CR) 10Umherirrender: [C: 031] zuul: archive the Wiki Object Model extension [integration/config] - 10https://gerrit.wikimedia.org/r/409851 (https://phabricator.wikimedia.org/T186516) (owner: 10MarcoAurelio) [20:45:34] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:53:11] !log deployed mobileapps@f14bdd5 to beta cluster [20:53:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:59:38] 10Project-Admins, 10MediaWiki-Documentation, 10Documentation, 10Pywikibot-Documentation, and 2 others: Consider archiving #MediaWiki-Documentation & #Pywikibot-documentation in favour of single #Documentation - https://phabricator.wikimedia.org/T142345#3965155 (10Dvorapa) [20:59:47] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965156 (10Dzahn) >>! In T182832#3965043, @mmodell wrot... [21:01:25] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-MarcoAurelio: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870#3965161 (10MarcoAurelio) >>! In T186870#3965005, @Huji wrote: > @MarcoAurelio what was the total execution time? Total row count is 128K I don... [21:02:25] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965163 (10Paladox) @Dzahn phabricator does not support... [21:10:44] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965181 (10mmodell) Yeah unfortunately 7.0 dropped a fe... [21:13:04] 10Gerrit: Cannot "Submit" on Gerrit, list of permissions says I should be able to - https://phabricator.wikimedia.org/T187120#3965191 (10MarcoAurelio) [21:19:59] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:20:39] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:20:41] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:20:50] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:21:14] PROBLEM - Puppet errors on integration-slave-docker-1002 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:25:40] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:26:33] who's taking care of those beta cluster puppet errors? [21:28:23] PROBLEM - Puppet errors on integration-slave-docker-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:30:44] hashar: for some reason jenkins is not picking https://gerrit.wikimedia.org/r/#/c/408065/ for gate-and-submit (yes, the extension has tests configured in zuul) [21:32:17] sorry, wrong patch: https://gerrit.wikimedia.org/r/#/c/409483/ is the right one [21:34:47] Hauskatze https://integration.wikimedia.org/zuul/ [21:34:52] seems to be listed there [21:35:20] I did 'recheck' paladox , but it should be on gate-and-submit [21:35:24] it's +2ed [21:35:34] but jenkins don't pick it for some reason [21:35:53] Hauskatze you need to re do +2 for it to re enter the gate and submit [21:36:09] Yaron_Koren did so [21:36:13] twice already [21:38:03] mwext-testextension-hhvm-jessie -> 'queued' heh [21:41:54] paladox: there's a dependency in that change for some reason [21:42:04] if we undo it, maybe it'll start gate-and-submit [21:42:16] Hauskatze ah that's why it didnt merge [21:42:19] https://gerrit.wikimedia.org/r/#/c/409309/ [21:43:09] yep, would rebasing break that dependency? [21:43:56] 16:18:48 Error: /home/jenkins/workspace/mwext-testextension-php55-jessie/src/extensions/PageForms/libs/jquery.rateyo.js should not be executable [21:44:05] Hauskatze maybe you could fix that ^^? [21:46:09] me? [21:46:15] how? [21:48:00] Hauskatze done [21:48:08] that's better :) [21:49:06] Hauskatze actually https://gerrit.wikimedia.org/r/#/c/409483/ can be abandoned [21:49:19] it's a blank commit [21:55:38] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [21:56:43] greg-g: how .. ahem... do i delete a page on office wiki? https://office.wikimedia.org/wiki/Tech_program_proposals/_(Part_of_tech_debt%3F)_Sunseting_and_Consolidation_Of_Systems [21:57:29] https://office.wikimedia.org/wiki/Tech_program_proposals/_(Part_of_tech_debt%3F)_Sunseting_and_Consolidation_Of_Systems?action=delete [21:57:36] (assuming you are a sysop there) [21:58:22] RECOVERY - Puppet errors on integration-slave-docker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:58:38] nuria_: I don't have the rights, but I know people :) [21:59:52] greg-g: boy the wikis [21:59:59] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [22:00:41] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:00:41] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:00:50] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:01:12] RECOVERY - Puppet errors on integration-slave-docker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:35] no_justification / greg-g you got mail [22:02:47] I saw. [22:11:21] (03Abandoned) 10MarcoAurelio: test [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/410002 (owner: 10MarcoAurelio) [22:28:49] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-MarcoAurelio: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870#3965367 (10MarcoAurelio) That said, on WMF production it has 2 long years of cleaning to do after we fixed the `$this->requireExtension` thing.... [22:45:51] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10User-Elukey: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3965408 (10Dzahn) There is T151070 (Experiment with php... [22:46:48] 10Phabricator, 10Operations: Upload php7.1 to apt.wm.org - https://phabricator.wikimedia.org/T160714#3108551 (10Dzahn) This issue is coming back on T182832 [22:48:45] 10Phabricator: Experiment with php 7.1 on phabricator - https://phabricator.wikimedia.org/T151070#2806532 (10Dzahn) T182832#3965043 now about upgrading to 7.1. experimenting early wouldn't have been bad after all. [22:49:41] 10Phabricator: Experiment with php 7.1 on phabricator - https://phabricator.wikimedia.org/T151070#3965427 (10Paladox) Yep. I've spun up a testing stretch instance for phabricator. I will create a task to add support for stretch within the phabricator class. [22:52:03] (03PS2) 10Legoktm: [WIP] Add mwext-phpunit-coverage-patch job [integration/config] - 10https://gerrit.wikimedia.org/r/408234 [22:52:07] 10Phabricator, 10Release-Engineering-Team, 10Operations: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#3965428 (10Paladox) [22:54:46] 10Phabricator, 10Release-Engineering-Team, 10Operations: Add support for stretch in the phabricator puppet class - https://phabricator.wikimedia.org/T187127#3965444 (10Paladox) p:05Triage>03High Needs to be high so we can try to resolve T182832 and T186620. [22:55:06] (03PS3) 10Legoktm: Add mwext-phpunit-coverage-patch job [integration/config] - 10https://gerrit.wikimedia.org/r/408234 [22:55:22] (03CR) 10Legoktm: [C: 032] Add mwext-phpunit-coverage-patch job [integration/config] - 10https://gerrit.wikimedia.org/r/408234 (owner: 10Legoktm) [22:56:36] (03Merged) 10jenkins-bot: Add mwext-phpunit-coverage-patch job [integration/config] - 10https://gerrit.wikimedia.org/r/408234 (owner: 10Legoktm) [23:03:29] (03PS1) 10Legoktm: Add "check coverage" to retrigger coverage checks [integration/config] - 10https://gerrit.wikimedia.org/r/410075 [23:03:31] (03PS1) 10Legoktm: Try out mwext-phpunit-coverage-patch with Linter [integration/config] - 10https://gerrit.wikimedia.org/r/410076 [23:03:41] (03CR) 10Legoktm: [C: 032] Add "check coverage" to retrigger coverage checks [integration/config] - 10https://gerrit.wikimedia.org/r/410075 (owner: 10Legoktm) [23:03:58] (03CR) 10Legoktm: [C: 032] Try out mwext-phpunit-coverage-patch with Linter [integration/config] - 10https://gerrit.wikimedia.org/r/410076 (owner: 10Legoktm) [23:04:47] (03Merged) 10jenkins-bot: Add "check coverage" to retrigger coverage checks [integration/config] - 10https://gerrit.wikimedia.org/r/410075 (owner: 10Legoktm) [23:05:10] (03Merged) 10jenkins-bot: Try out mwext-phpunit-coverage-patch with Linter [integration/config] - 10https://gerrit.wikimedia.org/r/410076 (owner: 10Legoktm) [23:06:29] !log deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076 [23:06:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:09:39] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10Patch-For-Review, 10User-MarcoAurelio: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870#3965485 (10MarcoAurelio) 05Open>03Resolved Well, the data is purged. I'll create another task so this script can run... [23:10:04] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10User-MarcoAurelio: Purge old IP data from AbuseFilter on the Beta Cluster - https://phabricator.wikimedia.org/T186870#3965488 (10MarcoAurelio) [23:30:30] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Phabricator email notifications should bundle events as the web interface does - https://phabricator.wikimedia.org/T85305#3965613 (10Aklapper) 05Open>03declined @Rical: Your comment is unrelated to the topic of this task. General comments... [23:35:59] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:37:21] PROBLEM - Puppet errors on deployment-redis05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:49:49] 10Beta-Cluster-Infrastructure, 10Operations: Remove video scaler instances from deployment-prep - https://phabricator.wikimedia.org/T187063#3965674 (10brion) I'm not using them for anything; should be clear to wipe them as long as the live servers serving the video scaler queues are not affected. :D [23:55:38] 10Gerrit, 10Release-Engineering-Team: Access to manage Gerrit repos to @MarcoAurelio - https://phabricator.wikimedia.org/T186690#3965676 (10demon) Ahhhh, [[ https://gerrit.wikimedia.org/r/Documentation/access-control.html#category_submit | found the issue ]]: > Submitting to the refs/meta/config branch is onl... [23:55:46] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Phabricator email notifications should bundle events as the web interface does - https://phabricator.wikimedia.org/T85305#943658 (10Tgr) Having digest emails would be a nice option for people with limited mental bandwidth for Phabricator noti... [23:56:33] 10Phabricator, 10Operations: Upload php7.1 to apt.wm.org - https://phabricator.wikimedia.org/T160714#3965680 (10mmodell) @MoritzMuehlenhoff: There is at least one 3rd party PHP 7.1 package available [[[ https://packages.sury.org/php/ | 1 ]]]. Could we not use their source packages to build our own binaries? I... [23:57:16] (03CR) 10Chad: [C: 032] Adding Sentry to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/409749 (owner: 10Chad) [23:57:56] (03Merged) 10jenkins-bot: Adding Sentry to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/409749 (owner: 10Chad)