[21:45:03] rfc meeting in 15 minutes [21:52:04] Brion which topic is today? [21:55:42] Zppix: topic is https://phabricator.wikimedia.org/T172165 (bump ver requirement to php 7.0) [21:57:04] Thanks [21:57:13] :) [22:00:01] #startmeeting TechCom RFC meeting about https://phabricator.wikimedia.org/T172165 | Wikimedia meetings channel | Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ [22:00:01] Meeting started Wed Nov 22 22:00:01 2017 UTC and is due to finish in 60 minutes. The chair is brion. Information about MeetBot at http://wiki.debian.org/MeetBot. [22:00:01] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [22:00:01] The meeting name has been set to 'techcom_rfc_meeting_about_https___phabricator_wikimedia_org_t172165___wikimedia_meetings_channel___please_note__channel_is_logged_and_publicly_posted__do_not_remove_this_note____logs__http___bots_wmflabs_org__wm_bot_logs__23wikimedia_office_' [22:00:11] * brion waves [22:00:23] Heya. [22:00:50] Ok, so RFC https://phabricator.wikimedia.org/T172165 has been discussed a bit before, there's been a lot of feedback :) [22:01:05] o/ [22:01:12] I believe current proposal is to require a minimum version of PHP 7.0 in MW 1.31 [22:01:27] which has some potential implications for making sure we've got things up to date in all production areas [22:01:50] :) [22:01:51] Do we have anybody repping the RFC? [22:01:51] <_joe_> brion: php 7.0 + HHVM [22:02:00] right -- php 7.0 or php7-equivalent-HHVM :) [22:02:15] or near-php7-equivalent-HHVM? [22:02:16] I don't think anomie is usually available at this time [22:02:19] <_joe_> not even, HHVM in its standard mode [22:02:20] but I can try and represent it [22:02:27] legoktm: thanks [22:02:44] "the subset of PHP7 that is supported by PHP 7.0 on Debian Stretch and HHVM 3.18.x" [22:03:08] #info current state: plan to support "the subset of PHP7 that is supported by PHP 7.0 on Debian Stretch and HHVM 3.18.x" [22:03:13] php4 should be good enough for everyone [22:03:25] php/fi was pretty awesome [22:03:31] <_joe_> bd808: well drop "on debian stretch", and also we could aim at any version of HHVM < 3.25 [22:03:31] perl 5 was great [22:03:43] So I think everyone is in favor of this plan, assuming it is possible right? No one is against bumping to 7.0 for 1.31? [22:03:44] scheduling for this meeting, we can pick two of any three of _joe_, legoktm and anomie [22:04:11] legoktm: I'm just a little worried about the timeline [22:04:14] my one concern is that folks are going to have a hard time _writing_ to that subset [22:04:25] did we have a plan on suitable linter settings etc? [22:04:30] brion: Writing, or testing to ensure they've written it? [22:04:32] or rather, any two of three [22:04:44] James_F: writing is easy, testing it works is hard ;) [22:05:02] no_justification: do you have specific timeline concerns? [22:05:03] Between phan and phpcs I think we can blacklist features that don't work - we already use phpcs to block some PHP7 features [22:05:12] Heres my concern have we tested to ensure we can support php 7 on CI and throughout mw core reliably ? [22:05:13] Like if the migration takes longer than anticipated (can/will happen), do we rush the migration? Hold 1.31.0? [22:05:23] Zppix: Yes. [22:05:27] One question that came up when we discussed this briefly in last week's techcom meeting was how powerful our CI is [22:05:35] Like, can we use phpcs+phan to enforce this subset [22:05:40] #info we don't appear to have a drive toward all-php7 for now. the hybrid is probably ok for 1.31 [22:05:58] for all of the main MW servers we already hit the needed software correct? [22:06:03] <_joe_> yes [22:06:04] #info concern about whether CI can support this combo and lint/test it reliably [22:06:09] wikitech and dumps are the lagging parts [22:06:10] RoanKattouw: We *should* be able to [22:06:11] Running unit tests on both PHP7 and HHVM is one thing (and probably a good idea), but our test coverage is not even kind of close to 100% [22:06:24] no_justification: One option would be to release 1.31 as 7.0+ but still keep master compatible with 5.x? [22:06:32] #info wikitech and dumps are lagging in debian version, others should be easy to update [22:06:47] <_joe_> The actual blockers in production are now blockers of the RfC ticket [22:06:48] #info 17:06:24 One option would be to release 1.31 as 7.0+ but still keep master compatible with 5.x? [22:06:53] That's an idea. [22:07:01] Bump the requirements for install/upgrade after branch [22:07:05] But master remains 5.x+ [22:07:09] full php7 is different than our current hhvm that supports some/many php7 features [22:07:17] #info Between phan and phpcs I think we can blacklist features that don't work - we already use phpcs to block some PHP7 features [22:07:26] legoktm: but in that release we should make it very clear we are bumping requirements? [22:07:37] Zppix: yes, obviously [22:07:40] legoktm: OK cool [22:07:51] the point of this would be to continue supporting snapshot* and wikitech? [22:08:21] and to give more time for php7 puppet work and major load testing I think? [22:08:24] The point would be so we can disentangle operational concerns from MW release cadence. [22:08:33] s/continue supporting/remove time pressure for migration of/ [22:08:34] That's my worry about timelines [22:09:00] #info point of hybrid mode is to disentangle operational concerns (updating server OSs) from MW release cadence for the moment [22:09:09] what's the hoped for date for 1.31? April 2018? [22:09:17] I've always been on board with PHP7+ bump, but I hate tying releases to our operational needs (cf my rants on tying wmf.X branch cycles to general release cycles) [22:09:20] <_joe_> I feel like we're circling back to discussions we had last week [22:09:20] May 2018. [22:09:26] Yeah, May-ish [22:09:35] <_joe_> sorry, some weeks ago [22:09:58] Cloud Services can commit to wikitech moving by then and also can help with migrating snapshot* [22:10:04] <_joe_> still, I am pretty confident we can have production ready for "hhvm + php 7.0 only" bt that date [22:10:11] #info [14:09:54] Cloud Services can commit to wikitech moving by then and also can help with migrating snapshot* [22:10:18] #info [14:10:00] <_joe_> still, I am pretty confident we can have production ready for "hhvm + php 7.0 only" bt that date [22:10:18] \o/ [22:10:21] <_joe_> I got confirmation from all concerned parties [22:10:33] _joe_: I don't doubt your confidence. I'm just wanting to remove the cross-team pressures. [22:10:46] Then we don't have to worry :) [22:11:05] Ops can do their thing, MW devs can do theirs. Everyone is happy af :) [22:11:30] <_joe_> no, both those migrations are about 2.5 years overdue already [22:11:39] <_joe_> that's us dragging everyone back. [22:11:43] it could all be done in a day of work if there wasn't so many things arbitrarily added into the dependency chain [22:11:47] not getting stuck support php5.5 for *years* beyond EOL is worth the near term pressure [22:12:02] if only we could compile mediawiki and php into a static binary that we deploy everywhere in one chunk (am i trolling or recommending docker? who can tell ;) [22:12:07] +1 to Tim, obviously. [22:12:20] <_joe_> TimStarling: what do you mean? [22:12:20] So is everyone in agreement to do this? [22:12:22] apparently to migrate wikitech off trusty, we need to migrate its database to the production cluster, but before we can do that, we have to do every other piece of pending database maintenance because that is more important [22:12:22] Im with getting back into latest stable versions asap [22:12:50] and apparently there's no point in migrating dumps to jessie, we can just migrate to stretch instead [22:13:04] but this will be the pilot deployment of MW on stretch, we need to build packages for that [22:13:13] oh fun [22:13:22] I think we have the hhvm packages now [22:13:28] * bd808 looks [22:13:31] <_joe_> TimStarling: if the timing doesn't match, people can just roll back to just upgrading to hhvm, I guess [22:13:38] #info some package work needed for wikitech migration? [22:13:41] <_joe_> bd808: dumps want to use php 7.0 [22:13:51] <_joe_> and there is *no* blocker to doing that that I know of [22:14:00] that is my plan, yes, after talking with moritz and others [22:14:08] <_joe_> apart adapting our puppet code, which would need to be done anyways for jessie [22:14:32] #info dumps needs update straight to stretch, which shouldn't be too awkward [22:14:40] So...is everyone in agreement to do this? [22:14:46] <_joe_> I think we should say "by date X, nothing in production that includes mediawiki can run on php 5.x" [22:14:57] <_joe_> and let the ops teams plan around that? [22:15:04] legoktm: sounds like :) [22:15:13] Any general disagreements with moving on with this? [22:15:17] Shall we put in last call state? [22:15:22] <_joe_> I would really much like to know what that X date should be [22:15:27] ah that's good yes [22:15:32] <_joe_> I heard different values [22:15:37] 2018-04-01 [22:15:42] <_joe_> ok [22:15:44] drop dead date [22:15:55] #info need to clarify the date? bd808 says 2018-04-01 as the drop-dead date [22:16:01] No disagreement here [22:16:05] great [22:16:08] but you know, the reason we want to go to PHP 7 is because we want to use PHP 7 features [22:16:16] ^ agreed [22:16:19] and if we can't meet that, our backup plan is to release 1.31 as 7.0+/hhvm with master still being compatbile with 5.x ? [22:16:25] true.... we need that second migration eventually [22:16:27] and we are talking about delaying that for 5 months for ops [22:16:31] hmmm [22:16:40] TimStarling: that dependency is silly and arbitrary- we could migrate the database on silver to stretch with no problem; and if you tell me moving to the main cluster is a huge blocker for us, we can drop other things [22:17:11] I was told it was ok to wait for s8 to finish the migration beecause it was not that urgent [22:17:13] moving the full fleet to php7 is more than a day [22:17:17] #info [tim] note we do want to actually *use* php7 features in prod. delay is not ideal, could be done differently if don't move wikitech db to main cluster [22:17:30] <_joe_> bd808: TimStarling is talking about turning off php5 [22:17:43] <_joe_> so that we can use php7 features that work with hhvm [22:17:47] <_joe_> the sooner, the better [22:17:48] ok I am exaggerating slightly, there are two reasons for wanting to migrate to PHP 7 [22:17:53] the other is PHP 5 EOL [22:18:12] the first is HHVM? [22:18:24] Well, that's why I liked legoktm's idea. Bump the requirements in the branch (for release purposes) but don't bump in master until production work is done [22:18:35] #info also php 5 EOL is scary [22:18:38] Then we /can't/ be tempted to use php7 stuff until time is ok [22:18:41] <_joe_> on the other hand, I don't see this as a good justification for having to duplicate work on the dumps hosts (migrating to jessie first, and migrating to stretch/php 7 in 6 moths of time) [22:18:58] bd808: the first reason is what I just said, using PHP 7 features [22:19:05] We NEED to get out of php 5 its seriously out of date esp security [22:19:23] Zppix: {{cn}} [22:19:34] We don't NEED to do anything [22:19:38] <_joe_> Zppix: that's an issue that the sysadmins running php5.x will have to solve [22:19:39] * no_justification puts his luddite cap on [22:19:58] _joe_: ah i misunderstood [22:20:04] I'm happy for the snapshots to be at the cutting edge of the curve, i.e. going first for stretch/php7 [22:20:16] that was my intent [22:20:24] not wait until 6 months and etc [22:20:29] I don't disagree that php5 (and especially 5.5) is aging, but we are not in a desperate security posture or we would not be talking about it on irc [22:20:56] <_joe_> bd808: the trusty EOL is in 2019, too [22:21:20] * bd808 owns 50% of the trusty hardware in prod and is aware [22:21:21] <_joe_> anyways, we're veering off-topic [22:21:32] Eh, point about migrations. We probably want deploy masters to be early on the upgrade list [22:21:49] * no_justification mutters stuff about linting and so forth [22:22:06] did we get a good idea about whether a suitable linter config can be set up? [22:22:28] <_joe_> yeah, CI is what is of more concern IMHO [22:22:31] wouldn't ci be sufficient even if the deploy masters don't get done? [22:22:42] apergos: nope. [22:22:43] only if test coverage is sufficient [22:22:48] which it probably ain't [22:22:53] Security patches. [22:22:59] Don't go thru CI [22:23:03] dang [22:23:09] heh [22:23:14] <_joe_> brion: well we have to guard against constructs that are php7.0 only, or are hhvm-only [22:23:15] brion: I can't think of any major problems with setting up linters [22:23:21] <_joe_> that can be done with a linter [22:23:29] legoktm is the relevant expert on linters [22:23:48] is there such a linter even? [22:23:49] #info legoktm says linting should be doable for what we need (checking incompatible constructs) [22:23:56] Yeah linters in CI shouldn't be hard, +1 to legoktm [22:24:25] MaxSem: mostly I think we can write some PHPCS sniffs (we already have some for banning PHP7 features) or a phan plugin if its more complicated [22:24:36] legoktm: That'd be very useful. [22:24:54] so yeah, no such linters:) [22:25:05] +1 legoktm [22:25:06] Linting in CI against hhvm/php7 only features seems doable (as well as against buggy hhvm versions of php7 features) [22:25:11] #info PHPCS sniffs should cover most of it [22:25:11] but what about prod linting? [22:25:17] We don't use phpcs in scap [22:25:19] Should we? [22:25:26] I was just thinking that [22:25:31] Is that the only way to ensure security patches wont break on some hosts? [22:25:35] Sounds like it. [22:25:44] Right now we shell out to php - l [22:25:57] Once we have the docker CI stuff more ready and the private jenkins set up, it should be much easier to do [22:26:01] That is insufficient against features hhvm thinks it can support but doesn't properly. [22:26:11] But that's a whole separate project to make security patches go through CI [22:26:13] #info scap may need better linting; example security patches go in manually and skip CI [22:26:20] Krinkle: Like JSON formatting. ;-P [22:26:58] I doubt that in the near term we are going to have security patches using exotic php7 features [22:27:16] <_joe_> bd808: indeed [22:27:21] And in the long-run we'll have only PHP7 in prod. [22:27:21] perfect is the enemy of good enough and all of that [22:27:26] So it doesn't matter. [22:27:39] * TimStarling imagines a world in which an emergency deployment to get the site back up is blocked on phpcs complaining about an "empty comment" [22:27:50] :) [22:27:56] Yeah, let's not do that. [22:28:22] OTOH, scap complaining about arrrrrrray would have kept the site up once. [22:28:24] TimStarling: This isn't about coding style part of phpcs [22:28:26] we'd use different rules [22:28:28] I imagine the phpcs rules we'd use in prod would be a much smaller subset [22:28:29] Yeah [22:28:41] This is about using phpcs's superior capacity compared to php -l to detect broken features [22:28:43] Basically lint, explicitly banned things, etc. [22:28:54] Which would lead to a commit trying to fix the site, to breaking it even more, if left unseen. [22:28:56] ok [22:29:53] TimStarling: "I know the site is down, but I didn't get the spacing around my parentheses correct!" [22:29:54] :p [22:30:08] OK, so there are several things to work on. Do we have a priority list so that people who can work on several know which to do first? [22:30:32] there should be items on the phab ticket [22:30:35] but not in order afaik :D [22:30:39] Yeah. [22:30:43] <_joe_> the ops side is already covered in subtasks [22:30:57] The phab subtasks are also not that clear because the dependency tree is unwieldy and hard to read [22:30:59] ok, any tasks like linting not already in there? [22:31:02] And we just invented scap-with-phpcs and add-php7-to-phpcs. [22:31:05] 1) Update prod hosts to HHVM 3.18 or PHP7 as needed 2) Scap using phpcs as linter, 3) Codesniffer for CI blacklisting PHP 7 features that are broken or absent in HHVM. [22:31:40] #action add linter config task to phab (scap-with-phpcs and add-php7-to-phpcs) [22:31:53] #info 1) Update prod hosts to HHVM 3.18 or PHP7 as needed 2) Scap using phpcs as linter, 3) Codesniffer for CI blacklisting PHP 7 features that are broken or absent in HHVM. [22:32:15] legoktm: you want to do the linter tasks (or at least add them to phab)? [22:32:24] James_F: phpcs for scap should be a pretty easy fix. [22:32:35] brion: sure, I can file tasks after this meeting [22:32:46] no_justification: Sure, I just worry that we take on too many things and don't make sure they're done. [22:32:51] thx [22:32:57] Well, scap one would fall to me or someone on my team :) [22:33:00] #info legoktm will set up the linter tasks [22:33:01] * no_justification puts on his releng hat [22:33:50] ok, any more specific task assignments we want to take out of here? [22:34:04] bd808 said the deadline for the ops tasks (snapshot* and silver) is 2018-04-01 [22:34:10] #info chad or someone on releng to set up the scap integratino of linting [22:34:27] TimStarling: making that sooner would be fine with me too [22:34:27] we good with that as drop-dead date? do we wnat a sooner target? [22:34:29] and I responded by giving the reasons I will be annoyed if it takes that long [22:34:37] :) yes [22:34:41] I don't know that I can commit to before mid January [22:34:55] early january will be busy, so i can see late jan or feb [22:35:09] my team has about 3 weeks of working time left before the end of the year [22:35:20] <_joe_> TimStarling: convince someone to hire more opsens then :) [22:35:25] dont' forget that late jan has a weak of dead time in it (meetings) [22:35:30] *week [22:35:37] _joe_: that's cheating! [22:35:40] february sounding better and better :D [22:35:42] Tbh, we could likely add the phpcs support right away, put it behind a config flag or something [22:36:44] if the work is scheduled for february, with april as a hard deadline, I guess that is OK [22:37:19] <_joe_> apergos: does that work for you? I guess we should anyways check with the team [22:37:31] <_joe_> I think the hard deadline is reasonable [22:37:47] it's fine, I would like to be done much earlier tbh [22:37:55] <_joe_> ok then :) [22:38:07] whee :) [22:38:23] apergos: might be easier to get dumps up and running on php7 on the early side of that [22:38:25] hopefully :) [22:38:41] #info provisional plan for work in february, with hard drop-dead date in april [22:39:26] ok, anything else to add or are we about wrapping up? [22:39:47] So we're good to move to last call? [22:39:51] <_joe_> +1 [22:40:01] the RFC description needs to be updated to reflect current consensus [22:40:12] brion: that's exactly the thought, it's why I want to just do it all in one step [22:40:47] anybody want to volunteer to update the task or sshall i attempt it from notes after this? [22:40:58] I'll do it [22:41:13] #action Tim to update task with current consensus; moving to last call [22:41:25] woot [22:41:40] ok thanks everybody, hopefully the moving parts will move in suitable sequence :D [22:41:52] any last-minute info points? [22:42:09] T178538 will be declined, right? [22:42:10] T178538: Bump PHP requirement to 5.6 in 1.31 - https://phabricator.wikimedia.org/T178538 [22:42:16] right [22:42:26] #info alternate T178538 will be declined [22:42:50] one last question [22:43:04] can we kill 5.5 compat right now? [22:43:15] No. [22:43:22] Prod has 5.5 boxes right now. [22:43:26] *cough* [22:43:29] (For the dumps and so on.) [22:43:30] not until those 5.x boxes die yeah :D [22:43:32] [2243][tstarling@silver:~]$ php --version [22:43:32] PHP 5.5.9-1ubuntu4.22 (cli) (built: Aug 4 2017 19:40:28) [22:43:45] aaa [22:43:55] I thought we're at least on 5.6:) [22:43:56] #info reminder the php 5.x boxen still in production until this migration is complete :) [22:44:00] MaxSem: If it makes you sad, help apergos and bd808. :-) [22:44:15] get me some mediawiki puppet manifests for stretch please :-) [22:44:24] James_F: gimme root or something? :P [22:44:30] though I thnk moritz is working on that [22:44:37] ok, i'm going to close out the meeting if no objection :D [22:45:06] #endmeeting [22:45:06] Meeting ended Wed Nov 22 22:45:06 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [22:45:06] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-11-22-22.00.html [22:45:06] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-11-22-22.00.txt [22:45:06] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-11-22-22.00.wiki [22:45:07] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-11-22-22.00.log.html [22:45:39] ok thanks all :) i'll write up the techcom radar mail later or tomorrow based on today's notes [22:51:08] talking about installs, and taking advantage of the many people still here… [22:51:14] who handles mathoid? [22:52:00] Physikerwelt with some assistance from Services [22:52:28] I think I'll bug him then [22:53:07] it doesn't seem to let itself be installed [22:59:37] cawling back to my normal channels, ttfn [22:59:40] *crawling