[00:40:58] chasemp: hi! If I want to test migrations on phab-01.wmflabs.org, can I use my own spage credentials or should/must I create a trellimport account? [00:41:24] for testing I would say whatever you like, but keep in mind most likely in prod you would be using a bot [00:41:34] and there are some lessons to be learned (like groups it may not be in) [00:41:49] so dealers choice but why not just make a quick bot ? :) [00:45:29] chasemp: I created https://phab-01.wmflabs.org/p/trellimport/ , I don't think I have rights to manipulate groups [00:46:24] most groups should be all users for edit i think [00:46:29] on phab-01 [03:18:38] 3operations, Phabricator: Create #site-incident tag and use it for incident reports - https://phabricator.wikimedia.org/T85889#956995 (10Aklapper) p:5Triage>3Low There's https://wikitech.wikimedia.org/wiki/Incident_documentation but also https://phabricator.wikimedia.org/project/query/all/?after=incid with t... [09:10:37] 3Wikimedia-General-or-Unknown, Code-Review: Implement a sane code-review process for MediaWiki JS/CSS pages on Wikimedia sites - https://phabricator.wikimedia.org/T71445#957068 (10Steinsplitter) >>! In T71445#956103, @Nemo_bis wrote: > I feel this report is //un cane che si morde la coda//. Unless someone summar... [14:16:20] Are any of the migration tools hardcoded to WMF Phabricator? [14:22:27] Lcawte: no [14:22:46] Lcawte: it's a 'bot' which interacts with the phabricator api, as far as I know [14:23:20] Is there any documentation other than the README file? [14:24:06] there's probably stuff on mediawiki.org [15:13:14] there really isn't much in teh way of docs other than teh readme no [15:13:44] a few people are using or have used the tools for reference or in general [15:14:09] most of it is pretty straight forward or really complicated and wmf won't ever do it again for 10 years so not much in teh way of docs [15:16:09] <^d> That's how migrations generally go. I found the same problem when we moved svn -> git [15:16:41] <^d> The internet is littered with abandonware conversion software because once an org finishes it, they're like "screw this" and don't ever have to deal with it again. [15:22:53] Interesting, I've found various tutorials for converting SVN repos to git that have all been useful? [15:23:57] <^d> Tutorials, sure. But the code they link to is mostly abandonware :) [15:26:01] yeah migrations pretty much always suck., [15:27:15] the other side to the coin is there is no cookie cutter migration [15:27:37] no tool will ever fulfill 100%, so it's always a tinker fest [15:28:26] <^d> yeppp [15:33:43] speaking of migrations, the upcoming phabricator changes are pretty wide-spread [15:34:13] when I ran the database migrations it did a lot of heavy-weight schema changes that take a long time with much data [15:35:17] fun [15:35:33] from what I gathered last night we are stalled on upgrading until you shake out the security extensino [15:35:42] so I figured wednesday was off the table [15:36:31] well I'm stuck in limbo [15:36:31] I have to either develop against the newest version or somehow rebuild my database and develop against an outdated schema [15:37:08] I figure it's more useful to develop against the new code and fix what's broken [15:38:11] we can do that but it will be a wild ride to update all at once [15:38:14] but I understand [15:38:25] it's probably most practical rather than fixing it all against a historical version [15:38:38] yeah wild ride indeed [15:38:56] it went through and changed the schema on every single table in every single database [15:39:03] awesome [15:39:06] yeah [15:39:23] I'm gonna guess that means hours of downtime [15:39:35] with the size of our data [15:40:15] honestly not sure, last large schema change was like 20 minutes? [15:40:15] but it wasn't //that// large [15:40:15] more than 30 minutes to be sure [15:41:27] I don't even know if it's necessary, it might be possible to skip some of the stuff in the migrations - it had normal schema changes and then it did some shit that changed the types of some columns to 'optimize' the database [15:43:13] I'm ok with you testing against HEAD or HEAD^ and then we can update and switch over to the event listener in one swoop [15:43:19] but it may be a long day for you if things go weird [15:43:25] but I don't see much of an alternative [15:44:46] yeah I think it's the only sane choice, rather than holding everything back [15:45:10] for who knows how long we'd be stuck on an old version - which might break the Sprint extension as well [15:46:05] there is a ticket or two out there from christopher kind of asking what the deal is with updating [15:46:21] could you ping him on what your doing so maybe you guys can sync up on an upstream commit to lock onto for upgrade [15:46:26] that way it's all nice and neat? [15:46:41] he's a bit hard to get ahold of (in this timezone maybe?) [15:48:03] twentyafterfour: is that the big utf8->bin change, so we can finally use emoji? ;-) [15:48:31] that should have happened already I thought [15:49:30] I'm not sure what it changed but it touched every single table in the db (or nearly every one) [15:50:08] <^d> fucking emojis. [15:50:20] 3Phabricator, operations: Edit policy should (almost) always be the same as view policy - https://phabricator.wikimedia.org/T85171#957474 (10chasemp) Is there anything actionable here? [15:50:52] stuff like this: [15:50:54] phabricator_worker lisk_counter counterName Better Collation Available, Better Character Set Available, Wrong Column Type [16:25:14] 3Phabricator, Phabricator.org: Users CCed in private tasks should be able to access them - https://phabricator.wikimedia.org/T518#957604 (10mmodell) I think I owe everyone some kind of progress report on this one: In the process of merging the completed changes (yesterday) with @chasemp, we ran into multiple is... [16:40:21] 3Phabricator, § Phabricator-Sprint-Extension: Create a continuous integration plan for Wikimedia Phabricator patches - https://phabricator.wikimedia.org/T85123#957644 (10mmodell) @christopher: I agree, we need to work out a way to coordinate our efforts so that upstream changes are minimally disruptive to each o... [16:48:06] 3Phabricator, operations: Edit policy should (almost) always be the same as view policy - https://phabricator.wikimedia.org/T85171#957664 (10Krenair) T787 is still unaddressed [17:25:00] 3Phabricator, Release-Engineering, § Phabricator-Sprint-Extension: Create a continuous integration plan for Wikimedia Phabricator patches - https://phabricator.wikimedia.org/T85123#957703 (10mmodell) [17:35:21] 3Phabricator, Release-Engineering, § Phabricator-Sprint-Extension: Create a continuous integration plan for Wikimedia Phabricator patches - https://phabricator.wikimedia.org/T85123#957724 (10mmodell) We discussed this today on the release engineering team meeting. Not much has been decided except that we should... [18:30:20] 3Phabricator, operations: Edit policy should (almost) always be the same as view policy - https://phabricator.wikimedia.org/T85171#957792 (10chasemp) 5Open>3Resolved a:3chasemp >>! In T85171#957664, @Krenair wrote: > T787 is still unaddressed Seems like @csteipp did so now: https://phabricator.wikimedia.o... [18:33:14] 3§ Phabricator-Sprint-Extension, Phabricator.org: Add Acceptance Criteria/Checklist feature to Phabricator tasks - https://phabricator.wikimedia.org/T1141#957803 (10ggellerman) I checked with the Analytics Research & Data team. For them, implementing checklists like Trello, while not a blocker for moving to Pha... [18:35:59] 3§ Phabricator-Sprint-Extension, Phabricator.org: Add Acceptance Criteria/Checklist feature to Phabricator tasks - https://phabricator.wikimedia.org/T1141#957806 (10mmodell) @ggellerman: no, and there isn't a dedicated team working on phabricator feature requests. Features like this are unlikely to happen withou... [19:13:02] 3Wikimedia-General-or-Unknown, Code-Review: Implement a sane code-review process for MediaWiki JS/CSS pages on Wikimedia sites - https://phabricator.wikimedia.org/T71445#957894 (10Aklapper) [Please do not add otherwise contentless "+1" comments as they create notifications for everybody. Use tokens or flags inst... [19:26:02] 3§ Phabricator-Sprint-Extension, Phabricator.org: Add Acceptance Criteria/Checklist feature to Phabricator tasks - https://phabricator.wikimedia.org/T1141#957934 (10Qgil) @mmodell, I don't think we can ask teams or anybody to provide development work for the features they request. What we can do is to ask the ri... [19:26:49] chasemp: howdy. I have commits making minor fixes to phabrictor/tools README and phabtools.conf.example, also Gilles Dubuc has mingleterminator.py and a .gitreview. https://gerrit.wikimedia.org/r/#/q/project:phabricator/tools,n,z for your consideration... [19:27:31] spagewmf: ah yes I'm actually 'rush' in gerrit :) [19:27:37] I did saw those in a cursory way a bit ago and forgot to mention [19:27:53] I think most of it looks ok but I am on ops duty this week and so a bit behind [19:29:08] 3MediaWiki-Core-Team, Security-Reviews, Phabricator: Aphlict security review - https://phabricator.wikimedia.org/T1286#957936 (10Qgil) There is ongoing work to get rid of Flash and use WebSockets. Should we resolve this task as Stalled in the meantime? See https://secure.phabricator.com/T6559 [19:29:32] ^ did see those I mean [19:29:55] ? Who is Chasemp in gerrit then? No big deal, just letting you know. *I've* figured out the host and certificate, but it'll help future generations of phab toilers [19:30:19] I have the username but it's just a shim to prevent someone from taking it in a confusing way [19:30:22] I know...super clear right [19:30:48] I got about a dozen accounts in a day and this all seemed reasonable then [19:32:29] [phab] `host = https://bugzilla.wmflabs.org/api/` confused me for a while :) [19:34:01] 3Phabricator, Collaboration-Team: Trello migration script - https://phabricator.wikimedia.org/T821#957958 (10Spage) >>! In T821#939498, @Qgil wrote: > Attachments and comments are clear requirements for a migration script, yes. Yup, those are what I'll work on after basic task creation. > Have you decided whi... [19:38:57] 3Phabricator, Collaboration-Team: Trello migration script - https://phabricator.wikimedia.org/T821#957973 (10Spage) [19:50:09] 3Phabricator, Release-Engineering, § Phabricator-Sprint-Extension: Create a continuous integration plan for Wikimedia Phabricator patches - https://phabricator.wikimedia.org/T85123#957995 (10hashar) Apparently we want to run a set of tests when a patch is proposed to the Sprint extension to ensure it is going to... [19:55:19] twentyafterfour: hi! I have no real idea how to test out Phabricator but added my thoughts on https://phabricator.wikimedia.org/T85123#957995 [19:55:39] twentyafterfour: i can probably sync with Christopher since we are in the same TZ and the wmde team has a large experience with our Jenkins / CI setup. [19:55:47] <^d> hashar: I have a phabricator role for vagrant :) [19:56:33] ^d: !!!! might want to share your experience on the task I linked so [19:56:45] cause I have Zero experience with phabricator [19:57:09] <^d> I've only touched like 4 files for elasticsearch stuff :p [19:57:47] <^d> You'd mainly just look at running `arc lint` like you do `phpunit` or `composer phpunit` or w/e [19:58:00] <^d> Arc is available from the arcanist repo in the ./bin dir [19:58:44] ^d: well the task is more about testing patches proposed to the phabricator extensions [19:59:11] at list I learned about the test command \O/ [20:00:55] hashar: I think the sprint extension doesn't even use gerrit code review? I may be wrong though [20:03:32] twentyafterfour: if so, no CI for them :-] [20:04:11] twentyafterfour: more seriously, there is a repo in Gerrit: ssh://gerrit.wikimedia.org:29418/phabricator/extensions/Sprint.git [20:04:28] so in theory, we could use branches / tag to track what is on Wikimedia [20:04:35] clone all repositories, checkout the deployed branch [20:04:41] and apply the Sprint proposed patchset [20:04:45] then run some magic test command [20:05:02] we currently have been using tags for deployment but I think a branch would be better perhaps [20:06:18] that would be a bit nicer [20:06:50] else a tag such as 'production' might work as well [20:07:23] hashar: well tags are not meant to be edited, so each release requires a new tag [20:07:44] +1 [20:07:46] you can't re-point a tag without violating the git phillosophy of tags and doing force-updates [20:08:09] I didn't want to sound annoying by asking a proper release branch to be cut [20:09:18] chasemp: can we modify gitdeploy to track a branch? perhaps the latest tag on that branch if we want explicit release versions... [20:09:18] we are going to want tags anyway for rollback tho? [20:09:41] yeah tags are nice for that [20:10:06] using just Head of a branch has a weird consequences for rollback and deployment etc, idk [20:10:20] I was thinking it would be nice if we could push a new extension update without requiring a patch to ops/puppet though [20:10:25] I saw the issue for ci with sprint app, and in general seems good to me but there are a few sharp edges I think [20:10:46] no I want to keep it under puppet control as long as it is maintained by ops [20:11:35] I definitely don't want any extension updating out from underneath me without going through puppet repo review, etc [20:11:44] if we had some CI and reasonable code review then we could deploy changes with some confidence that we aren't about to break production [20:12:10] yes but the mass of operations data there [20:12:15] contract numbers, serial numbers, contact info [20:12:40] it's just a friendly environment for continuous integration between componenents at this point [20:12:49] your right maybe we could get there, idk haven't thought much about it really [20:13:02] not a friendly environment I mean :) [20:13:44] for me, now since I'm clearly stuck deploying it I'm not going to roll out a new version of sprint app without making sure it doesn't allow some tomfoolery with permissions because it's doing some (while not sketchy) things taht are dangerous [20:14:34] we need like a metric ton better testing and process to do that stuff I think [20:14:45] could an extension potentially hijack / workaround the permissions? [20:14:53] yes [20:14:59] well the present issue is, the HEAD of sprint repo requires a very new version of phab which we don't have in production [20:15:26] hashar: an extension can do nearly anything, there is no sandbox [20:16:20] so we need review and +2 by trusted folks [20:16:28] yes [20:16:34] yes and for now it has been a solo mission [20:16:35] which is what we are doing for mediawiki [20:16:40] almost none of the sprint app is reviewed really well [20:16:44] I do a functional review [20:16:46] when I update [20:16:46] the mw extensions have +2er as well most are volunteers [20:16:54] to make sure it's not doing anything seriously bad [20:17:22] as I understand Phabriactor maintenance is mostly you two [20:17:29] which I'm fine for now but yeah part of the problem is [20:17:37] would be nice to figure out a way to scale out review to a few more folks [20:17:37] sprint app is it's own thing over there --> that has little review ongoing [20:17:45] (even if final approval / +2 ends up in your hands) [20:18:55] this is all a work in progress for sure :) [20:19:05] but in general there are 3 repo's for phab that need to be in sync [20:19:17] and I like tagging them in a predictable way tied back to an upgrade issue that links incoming fixes [20:19:19] that can more or less be enforced by CI + tests [20:19:22] and it provides reasoable rolle back [20:19:33] and the other issue is we have the security extension which also depends on upstream version changes, and it has to be maintained in sync with the sprint extension [20:19:56] what is the hiccup with sprint vs security extensions or is it just [20:20:00] they need to agree on a version fo phab [20:20:01] chasemp: rollback becomes a problem with database schema changes [20:20:06] so there are four repos to keep in sync / test together rights? phabricator / libphutil / Sprint / security ? [20:20:14] once you deploy the recent upstream changes, there is no rollback [20:20:14] yes it does but I always do a backup before doing a roll forward [20:20:22] so taht + tag change in puppet [20:20:28] is supposed to be a full restore [20:20:37] hashar: yes those repos, plus maybe arcanist too? [20:20:52] I suppose you could say yeah all of them need to be in agreement [20:21:23] so if we had a release branch on each repo [20:21:28] phabricator and libphutil must always be in sync to some point-in-time, so whatever is the current HEAD at the time we choose to branch it for a release [20:21:32] alternatively, you could use submodules, and tell gerrit not to magically track them [20:21:47] I really don't want to use submodules tho [20:21:50] life is hard enough as it is [20:21:56] and then the extensions have a dependency on that version (to some lesser extent than the dependency between libphutil and phab) [20:22:07] whenever someone propose a patch on one of those repo against the 'release' branch, we can make Zuul to checkout the 'release' branch on each of those repo, then apply the proposed patchset and run the tests. All the logic is already available [20:22:09] extensions can and do span version fo phab [20:22:13] it's only if there is a breaking change [20:22:16] which can happen or not [20:22:38] hashar: that's well and good but what tests would you run? [20:22:42] I'm not understanding that part [20:22:43] then to deploy, one just checkout the release branches of all repos, which have passed tests together [20:22:58] hashar: that seems pretty good [20:23:01] chasemp: ah hmm. I am assuming there is some decent level of test coverage [20:23:15] afaik that is not the case [20:23:25] with the additional step of ... tagging the tip of release branches for release checkpoint/rollback [20:23:34] for mediawiki, when someone propose a test against the REL1_24 branch of an extension, it is tested using mediawiki/core REL1_24 branch as well. Works well to prevent breakages [20:23:38] if you wanted to keep your version in a branch [20:23:41] (we do now already) [20:23:46] hashar: we've got zero test coverage currently [20:23:49] and then tag it before deploy and we make puppet match [20:24:07] sure that makes sense [20:24:07] we ALREADY use a branch now :) [20:24:07] twentyafterfour: so there is little we can do CI wise :-( [20:24:08] because we are maintaining the local oauth stuff [20:24:17] or at worth we could do some Selenium functional tests (*evil*) [20:24:29] yeah so what I have seen is no tests so i was wondering where this was going [20:24:35] like i said, we are leagues away from this [20:24:39] being super viable [20:24:45] and no one has even talked to christopher yet [20:25:11] hashar: well we need the testing infrastructure and we need to make the tests, chicken-egg problem, but having the infrastructure even with minimal tests would be good start [20:25:19] that is true [20:25:30] and it would be a good selling point to get sprint app to conform [20:25:45] so lets build the test job infra [20:25:52] but I don't have a really clear idea on what testing woudl even make sense :) [20:25:55] using it as a foundation to start getting test [20:26:02] keep in mind that it was christopher who proposed that we need a release process, I think he is on board already [20:26:11] cause writing tests that are not exercised automatically makes little sense to me [20:26:21] he'll have to stop doing self +2's [20:26:32] that is workable [20:26:35] say one thing and doing another in my mind right now [20:26:43] if I had +2 on that repo then he could have me do reviews [20:26:56] the devs at WMDE have a good culture of tests. I guess he self +2 because he lacks other reviewers / devs in the project [20:27:13] let's approach that then tentatively and try to get a test going and see how it works [20:27:22] but yeah we can use tags for tracking in conjunction with a deploy branch [20:27:24] I've had trouble finding anyone to review my code before, I just never thought self +2 was something that wouldn't get me fired ;) [20:27:25] as we basically do that now [20:27:29] git checkout tags/foo [20:27:33] works no matter what branch the tag is in [20:27:51] that is a good start :] [20:28:05] ok wlel thanks for getting involved hashar :) [20:28:26] so if one of you could reply on https://phabricator.wikimedia.org/T85123 with step by step instructions that would be nice [20:28:32] I am looking for repositories to clones [20:28:37] their destination [20:28:49] twentyafterfour: do you want to make our current deploy branch something predictable like "production" [20:28:50] or something [20:28:53] and an idea of a basic test command to run :] [20:29:20] chasemp: production works [20:31:34] does phabricator uses PHP 5.3 ? [20:31:45] production branches in all repo's and then a tag to match across the main 3? [20:31:46] hashar: yes I believe so [20:31:51] whateer trusty is on [20:32:11] PHP 5.5.9-1ubuntu4.5 (cli) (built: Oct 29 2014 11:59:10) [20:32:54] 3Phabricator: getting PhabricatorDataNotAttachedException error - https://phabricator.wikimedia.org/T85950#958083 (10Jaredzimmerman-WMF) 3NEW [20:33:54] chasemp: that is interesting :D [20:34:07] I made the CI Trusty instances to use HHVM by default iirc [20:34:40] $ php --version [20:34:40] HipHop VM 3.3.1 (rel) [20:34:47] hashar: phabricator doesn't support HHVM officially [20:34:52] yeah I can imagine [20:46:08] * hashar whistles [20:47:29] 3Phabricator, Phabricator.org: getting PhabricatorDataNotAttachedException error - https://phabricator.wikimedia.org/T85950#958142 (10Krenair) [20:48:27] twentyafterfour: chasemp: so at least arcanist seems to work with php 5.3.x :] https://integration.wikimedia.org/ci/job/phabricator-wmf-integration/3/console [20:49:30] phabricator is backwards compatible to 5.2 I believe [20:49:43] that job above clones a bunch of repositories including arcanist [20:49:46] all at master branch [20:49:57] then find the .git directories and runs arc lint :] [20:50:54] nice [20:51:11] this way we have ONE test :] [20:57:41] 3Wikimedia-General-or-Unknown, Code-Review: Implement a sane code-review process for MediaWiki JS/CSS pages on Wikimedia sites - https://phabricator.wikimedia.org/T71445#958176 (10MZMcBride) >>! In T71445#957894, @Aklapper wrote: > [Please do not add otherwise contentless "+1" comments as they create notificatio... [22:16:58] 3Phabricator.org: Phabricator tags should be listed as tags, not projects - https://phabricator.wikimedia.org/T75961#958312 (10Nemo_bis) >>! In T75961#943723, @Nemo_bis wrote: > An easy solution would be to rename the "project" type (the default) to "component", a terminology users will be familiar with. [[http...