[00:17:06] 10VPS-project-codesearch, 10Operations: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199 (10Dzahn) I am for this. This is actually pretty easy to do since we already puppetized it. If we just create a ganeti VM and put the puppet role on it then all that is left should be basical... [00:32:03] 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T263184 (10dancy) @Krinkle deployed the fix for T267668 to wmf.16 and testing looks good. group0: wmf.18 group1: wmf.16 group2: wmf.16 @greg says we c... [00:32:41] 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release, 10Train Deployments: 1.36.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T263184 (10dancy) [01:36:58] 10Release-Engineering-Team (Pipeline): pipelinelib should automatically update image versions Cloud VPS nodes that use profile::docker::services - https://phabricator.wikimedia.org/T267975 (10jeena) My thoughts are that we could add an update script similar to the one in the deployment-charts repo to the cloud/i... [02:36:35] !log dannys712@deployment-deploy01:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php metawiki DannyS712 [02:36:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:26:30] 10Gerrit, 10Developer Productivity, 10User-DannyS712: Include gate pipeline test results in the test results display - https://phabricator.wikimedia.org/T268307 (10DannyS712) [04:15:13] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10GitLab, 10Documentation, 10User-brennen: Document GitLab Development Kit setup on-wiki - https://phabricator.wikimedia.org/T268269 (10Reedy) [04:59:55] 10Gerrit, 10Security-Team, 10Security: Gerrit security advisory 2020-11-17 - https://phabricator.wikimedia.org/T268009 (10thcipriani) >>! In T268009#6634679, @hashar wrote: > Gerrit has been upgraded. Next thing is to find out whether we can drop the work around we have applied to All-Projects / All-Users.... [05:07:58] (03CR) 10Jay (CIS-A2K): "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/642097 (https://phabricator.wikimedia.org/T267488) (owner: 10Jay (CIS-A2K)) [05:09:12] (03CR) 10Jayprakash12345: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/642097 (https://phabricator.wikimedia.org/T267488) (owner: 10Jay (CIS-A2K)) [05:12:02] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [06:13:02] James_F about https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GlobalWatchlist/+/642090 is it worth starting a discussion somewhere? [06:13:16] Eh. [06:13:23] If it comes up again, maybe. [06:13:42] The SIP covering prod code was about "don't emit hard-deprecation warnings in prod, you'll melt the servers". [06:13:48] * James_F shrugs. [06:14:02] My first version of the patch tried to say that the entire extension was @internal and not a stable interface, but Urbanecm thought that wouldn't be okay... [06:14:09] I don't have the energy to try to run a TechCom RfC for months just for a two word understanding. [06:14:22] yeah, makes sense [06:14:32] thanks for the quick review though [06:15:17] The very best code review happens after 22:00. :-) [06:16:05] what timezone are you in? I find my best code review happens earlier in the day than that [06:16:10] PST. [06:16:33] My best "in depth" code review is before 10:00 local, yes. [06:16:50] But unless you're writing multi-hundred line patches, CR isn't too much work. [06:16:57] And if you are, please split them up. ;-) [06:17:42] what about multi thousand? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/600426 :) Already split up into like 20 patches that have merged, and a bunch to go that haven't yet [06:18:01] Yeah, I've been carefully pretending I've not seen that one. ;-) [06:18:06] Amazing work, BTW [06:19:16] thanks - currently merged work can be seen at https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/heads/master/includes/editpage/Constraint/ and EditPage::internalAttemptSave [06:19:20] The patch-I'm-most-dreading-and-looking-forward-to-merging is https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/628133 [06:19:40] I particularly like UnicodeConstraint. [06:20:05] Having implemented the equivalent string injection in VE to make things work. [06:20:23] Allowing VE to use an EditManager and not re-create EditPage instances will be wonderful. [06:20:28] lol, that abuse filter patch is "only" ~3 thousand lines of code [06:20:36] Yeah. :-) [06:20:40] Well, for VE to not use an edit page, you don't need to wait for EditManager [06:20:49] https://codesearch.wmcloud.org/search/?q=%E2%84%B3%F0%9D%92%B2%E2%99%A5%F0%9D%93%8A%F0%9D%93%83%F0%9D%92%BE%F0%9D%92%B8%E2%84%B4%F0%9D%92%B9%E2%84%AF&i=nope&files=&repos= [06:22:05] (at least in part) - one of the uses (https://gerrit.wikimedia.org/g/mediawiki/extensions/VisualEditor/+/6a238b784e2f12cab85a19b46839e14252496caf/includes/ApiVisualEditor.php#531) can be removed once T252962 is done [06:22:06] T252962: Move "templates on this page" logic out of EditPage - https://phabricator.wikimedia.org/T252962 [06:22:16] Nice. [06:22:36] We should make a task for that migration in VE, blocked by that work. [06:22:39] Eh, effort. [06:23:06] eventually; there is still so much to do [06:24:20] Yeah. [06:25:06] also, I added the example at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/642096 [06:25:59] DannyS712: Thanks, C+2'ed. [06:26:32] :) [06:26:32] Who knew that the secret to getting fast code reviews was just to ask James_F to review? :) [06:26:39] Umm. [06:26:44] Too many people. ;-) [06:27:44] well, I won't push my luck tonight then [06:27:56] * DannyS712 is also offering fast code reviews if anyone wants [06:31:52] how efficient is it if I want to query the logging table for entries with a specific log_comment_id ? [06:32:12] Not sure, but I imagine it's not indexed on that. [06:32:15] So "very bad". [06:32:21] The log comment split out is new. [06:32:37] "New" === "in the last five years". ;-) [06:32:41] * DannyS712 adds the log_type, log_action, and log_actor conds too, and then its really fast [06:33:25] Yeah: https://gerrit.wikimedia.org/g/mediawiki/core/+/b1796f314e48e7f6a02a75d838cb8637aa866891/maintenance/tables.sql#939 [06:33:26] context: WMFOffice locks accounts by referring to internal global ban numbers, wanted to know who `#00028530` was, so searched for other locks by the same staff account with the same reason [06:33:49] I guess I could also set log_namespace = 2 [06:33:59] * James_F shrugs. [06:34:01] Sorry. [06:34:24] for what? Everything works :) [07:13:15] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics, 10Patch-For-Review: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10elukey) We have the following config on all stat nodes: ` elukey@stat1008:~$ cat /etc/gitconfig # vim: set ts=4 sw=4 et: # Thi... [07:49:10] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [07:50:58] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris) [08:14:44] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10elukey) @brennen can you retry and see if it is fixed? :) [09:07:25] 10Gerrit, 10Developer Productivity, 10User-DannyS712: Include gate pipeline test results in the test results display - https://phabricator.wikimedia.org/T268307 (10QChris) I don't have much opinion on which test results to include. @Krinkle suggested on https://gerrit.wikimedia.org/r/c/operations/puppet/+/60... [09:19:59] 10phan-taint-check-plugin: Make caused-by lines longer - https://phabricator.wikimedia.org/T257350 (10Daimona) 05Open→03Resolved [09:20:01] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [09:23:36] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10kostajh) >>! In T268290#6635934, @elukey wrote: > @brennen can you retry and see if it is fixed? :) Hmm, now I get: ` kharlan@stat1008:~$ git clone... [09:24:29] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10kostajh) Oops, sorry the HTTPS clone was when I was experimenting with that instead of the SSH clone. With ssh clone, I still get a timeout: ` khar... [09:30:25] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10elukey) @kostajh sorry wrong ping in Phab :) You are definitely right, I added the IPs related to gerrit1001 and gerrit2001, but apparently it is no... [09:32:34] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet Failures on deployment-cache-upload06 - https://phabricator.wikimedia.org/T261513 (10hashar) [09:33:14] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-Ryasmeen: Beta needs to be upgraded to Varnish 6 - https://phabricator.wikimedia.org/T267561 (10hashar) [09:33:45] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet Failures on deployment-cache-upload06 - https://phabricator.wikimedia.org/T261513 (10hashar) Most probably that was due to T267561, the instance were running Varnish 5 while Puppet deployed VCL files intended for Varnish 6. That probably has been broken for more... [09:35:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations: Puppet failure on deployment-cache-text06 - https://phabricator.wikimedia.org/T256064 (10hashar) 05Open→03Resolved That must have un fixed at some point. It is no more happening. [09:38:03] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10hashar) 05Open→03Resolved **Thank you to everyone that acted on this task.** All the instances mentioned in this task are passing fine now! There are a couple more breakage but they can be acted on i... [09:38:34] 10Beta-Cluster-Infrastructure: Puppet failures on many hosts - https://phabricator.wikimedia.org/T267006 (10hashar) [09:48:24] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10JavaScript, 10Patch-For-Review: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10hash... [09:58:30] 10Gerrit, 10Operations: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs - https://phabricator.wikimedia.org/T165631 (10hashar) [10:06:05] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics, 10Patch-For-Review: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10elukey) @kostajh should be better now, can you retry? [10:16:08] 10Gerrit, 10Operations: move gerrit.wm.org SSH service to private/behind LVS like phab-vcs - https://phabricator.wikimedia.org/T165631 (10ayounsi) Decent transition might be to: # Create public LVS VIP # Test it on the Gerrit instance (in public) # If all good, decom the current (non-LVS) VIP Later one: # Move... [10:29:20] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10kostajh) >>! In T268290#6636219, @elukey wrote: > @kostajh should be better now, can you retry? Yes! Now I just need to figure out the key issue. `... [10:56:06] 10Project-Admins: New project: Wiki-Techstorm-2021 - https://phabricator.wikimedia.org/T268223 (10Aklapper) @Ecritures: Hi, where to find info about a 2020 addition? Asking as https://www.mediawiki.org/wiki/Wiki_Techstorm does not mention it... Also, what should happen with the [75 open tasks from 2019](https:/... [10:57:14] 10Project-Admins: New project: Wiki-Techstorm-2021 - https://phabricator.wikimedia.org/T268223 (10Aklapper) 05Open→03Stalled [10:59:37] 10Project-Admins, 10Platform Engineering, 10User-DannyS712: Create #mediawiki-namespaces - https://phabricator.wikimedia.org/T249998 (10Aklapper) @DannyS712: Do you know? (In any case, tagging #Platform_Engineering if they also see value in this - if not we might want to improve existing project description... [11:13:31] 10Project-Admins: New project: Wiki-Techstorm-2021 - https://phabricator.wikimedia.org/T268223 (10Ciell) Hi, thank you for the request Ecritures, good idea! Would it make more sense to call it 'Techstorm 2021' maybe? ;) [11:18:12] 10Project-Admins, 10Maps: Create an #OSM tag to identify tasks with OpenStreetMap relevance - https://phabricator.wikimedia.org/T207017 (10Aklapper) I'd love to have input from #Maps folks if this would help in managing / finding tickets. Plus I'd like to know where to [file upstream](https://www.mediawiki.org... [11:21:42] 10Project-Admins: Tag:#KB-TeamWiki - https://phabricator.wikimedia.org/T268224 (10Aklapper) 05Open→03Stalled Hi @DanielleJWiki, thanks for taking the time to report this (and for the interest in using Wikimedia Phabricator)! I don't understand the difference to T268222 here (a project is a project tag, and... [11:26:02] 10Project-Admins: Project (+workboard) for Koninklijke Bibliotheek (National Library of The Netherlands) - https://phabricator.wikimedia.org/T268222 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #Koninklijke-Bibliotheek-Team-Wiki has been created: https://phabricator.wikimedia.org/pr... [11:27:29] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10kostajh) OK, this now anonymous HTTPS clone works: ` lang=bash git clone "https://gerrit.wikimedia.org/r/research/mwaddlink" && (cd "mwaddlink" && m... [11:27:39] hi o/ - I somehow managed to push a tag (upstream/3.4.1) to gerrit (https://gerrit.wikimedia.org/r/admin/repos/operations/debs/helm3) that does not appear on new clones, gitlies or in gerrit UI. I can not delete the tag (internal error) and I can not force-push it (failed to lock). Can someone help me out of this situation, please? :) [11:56:32] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Developer-Advocacy, 10Wikimedia-Tech-Talks: Tech Talks Proposal: new CI system candidate demonstration - https://phabricator.wikimedia.org/T220695 (10Aklapper) 05Stalled→03Open This task isn't [stalled](https://www.m... [12:12:40] 10VPS-project-codesearch, 10TechCom: Automatically index extensions in CodeSearch - https://phabricator.wikimedia.org/T268328 (10daniel) [13:01:57] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Developer-Advocacy, 10Wikimedia-Tech-Talks: Tech Talks Proposal: new CI system candidate demonstration - https://phabricator.wikimedia.org/T220695 (10LarsWirzenius) Adding Tyler and Brennen as subscriber, they're on top... [13:06:52] (03PS1) 10BBlack: operations-dnslint: create version 0.0.9 [integration/config] - 10https://gerrit.wikimedia.org/r/642392 [13:06:54] (03PS1) 10BBlack: operations-misc: bump dnslint to 0.0.9 [integration/config] - 10https://gerrit.wikimedia.org/r/642393 [13:08:50] (03PS1) 10Lars Wirzenius: Bump version to 3.16.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/642394 [13:09:29] (03CR) 10Lars Wirzenius: [C: 03+2] "Self-merging as part of release process." [tools/scap] - 10https://gerrit.wikimedia.org/r/642394 (owner: 10Lars Wirzenius) [13:12:45] jayme: Should I simply delete the "upstream/3.4.1" tag? [13:13:39] qchris: Don't know but lets try. I will push it again afterwards and we see if I do better this time :) [13:14:24] I checked, and it replicated to both gitiles and github [13:14:33] https://github.com/wikimedia/operations-debs-helm3/tags [13:14:37] https://gerrit.wikimedia.org/g/operations/debs/helm3/+/refs/tags/upstream/3.4.1 [13:15:09] last one gives me Cannot parse URL as a Gitiles URL [13:15:22] and the tag is not visible (for me) at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/helm3/ [13:15:35] That might be related to the security issue mitigation from a few days ago. [13:15:59] Let me check if that mitigation is still in place. [13:16:22] Yup. The mitigation is still in place. [13:16:33] So it's expected that you cannot see the tag it gitiles. [13:16:42] But the tag is there. [13:17:08] And once the mitigation goes away (that will happen soonish), you'll be able to see the tag in gitiles as well. [13:18:53] Does that work for you or should we still remove the tag (pushing it afresh won't change the fact that you won't see it in gitiles) [13:19:30] hmm...if I clone a fresh copy of the repo I don't get the tag as well [13:19:41] Yup. That's expected as well. [13:19:55] As long as the mitigation is in place, you will not see the tag. [13:19:58] oh...thats pretty bad as builds don't work then :) [13:20:35] But if the mitigation is going to be removed soon, I'm okay with that [13:20:59] IIRC they said they'll remove it today. [13:21:37] okay, cool. Thanks a lot for looking into it! [13:22:10] yw. [13:46:09] 10Beta-Cluster-Infrastructure: New Scap release candidate .deb not visible on deployment-cumin - https://phabricator.wikimedia.org/T268337 (10LarsWirzenius) [13:56:40] 10Gerrit, 10Developer Productivity, 10User-DannyS712: Include gate pipeline test results in the test results display - https://phabricator.wikimedia.org/T268307 (10Krinkle) The "Gate pipeline build" is a superset of "Main test build" and only appears after the C+2 event. I think it's a good idea indeed to i... [13:58:37] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10elukey) >>! In T268290#6636347, @kostajh wrote: > OK, this now anonymous HTTPS clone works: > > ` lang=bash > git clone "https://gerrit.wikimedia.or... [14:10:25] 10MediaWiki-Releasing, 10MW-1.36-release, 10MinervaNeue (Tracking), 10Readers-Web-Backlog (Tracking): Bundle Minerva Neue skin with MediaWiki - https://phabricator.wikimedia.org/T191743 (10Aklapper) [14:10:28] 10MediaWiki-Releasing, 10MW-1.36-release, 10MobileFrontend (Tracking), 10Readers-Web-Backlog (Tracking): Bundle MobileFrontend extension with MediaWiki - https://phabricator.wikimedia.org/T191734 (10Aklapper) [14:10:44] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Tracking-Neverending: Expand the set of bundled extensions and skins to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349 (10Aklapper) [14:26:22] hashar: wanted to ping you one this https://gerrit.wikimedia.org/r/c/operations/puppet/+/638678 i think you have looked at doing something simlar to this in the past [14:27:22] jbond42: possibly yes maybe? :] [14:28:12] jbond42: I am unlikely to dig into rspec-puppet material anytime soon :-\ [14:29:34] hashar: no problem it needs review from someone in service ops, i thught you had some simlar patches i.e. to have our spec tests use the real hieradata directory as well as the private repo and dropping the need to specify dependencies etc [14:29:48] more just wanted to bring it to you attention so you know its there :) [14:30:24] in theorey we should be able to test a full role using puppet-rspec with very littel effort now (once this is merged) [14:38:24] jbond42:I might have wrote some kind of whole scale integration test suite based in a /spec repository at the root of puppet.git [14:38:30] jbond42: but that was ages ago I guess [14:39:00] I think rspec-puppet can be made to load our hiera file as well, though the suite has to generate the hiera config file from our erb template [14:39:14] but yeah hmm, I cant realistically look into it [14:41:16] yes that is that rake_modules/spec_helper.rb does but no problem like i said just more letting you know it exists [14:42:35] 10VPS-project-codesearch, 10TechCom: Automatically index extensions in CodeSearch - https://phabricator.wikimedia.org/T268328 (10Ladsgroup) Codesearch searches for extensions and rebuild it on daily basis: https://gerrit.wikimedia.org/g/labs/codesearch/+/refs/changes/47/641847/1/write_config.py#35 Do you mean... [14:56:26] 10VPS-project-codesearch, 10Operations: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199 (10Ladsgroup) If it's that simple and a ganeti VM makes things much easier, I think doing it would be pretty easy. 1- We need to use webproxy for pulling from github 2- it needs a security review? [15:05:39] hashar: About the workspace status script for building Gerrit ... was the branch name on your local machine `stable-3.2` when building, or something else? [15:06:18] qchris: wmf/stable-3.2 ;) [15:06:37] Ah! That explains things. [15:06:42] which the python workspace_status_release.py script obviously does not recognize hehe [15:06:50] so eventually I have build it without the helper [15:07:21] Yup, I saw that in backlog. That's why I'm asking. [15:07:37] 10Release-Engineering-Team, 10serviceops, 10Security, 10cloud-services-team (Kanban): Implement SSH CA (certificate authority) for host keys? - https://phabricator.wikimedia.org/T268344 (10LarsWirzenius) [15:08:06] For the plugins, we really want to use that script. But for the main war, ... meh. As long as the version name is good enough (which it certainly is right now). [15:08:34] I see you even updated the wiki accordingly. Thanks! [15:09:15] and we have some plugins forked on gerrit [15:09:20] to further complicate things ;] [15:09:44] Yes :-( Like the zuul plugin is unmaintained. [15:09:59] and no one upstream cares about the gitiles font loading. [15:10:32] well gitiles hmm [15:10:39] it could use a shit ton of polishing [15:10:50] such as: it has a lot of various config settings that are not configured [15:10:56] Yup. [15:11:47] But since our future is gitlab, I guess these things are now worth the time. I mean ... the number of Gerrit upgrades we'll do is counted :-) [15:12:27] and do not support forging canonical urls using gitiles.conf: gerrit.baseUrl = https://gerrit.wikimedia.org/g [15:12:44] 10Release-Engineering-Team-TODO: Hosting wikibase release tarballs on releases.wikimedia.org - https://phabricator.wikimedia.org/T268345 (10toan) [15:13:15] * qchris sighs [15:13:34] then [15:13:35] 10Release-Engineering-Team-TODO: Hosting wikibase release tarballs on releases.wikimedia.org - https://phabricator.wikimedia.org/T268345 (10toan) [15:13:51] it is not like I know anything about java or having to use an IDE :D [15:14:43] IDEs? ... pfff ... https://xkcd.com/378/ [15:15:47] yeah well [15:15:56] not sure vim is a pleasant experience for java development [15:16:18] maybe I should take a sabbatical and finally learn java. I am a couple decades outdated [15:16:34] Not sure there is really any pleasant experience for java development these days. [15:17:15] Meh. I digress. [15:17:53] Glad we're finally on 3.2.5 :-) [15:18:45] next: switch to java 11 and upstream .war [15:18:52] then I guess 3.3 [15:19:06] Looking forward to these changes! You rock! [15:19:27] yeah don't hold your breathe though [15:19:48] :-) [15:20:43] http://gerrit-documentation.storage.googleapis.com/Documentation/3.3.0/user-attention-set.html is super interesting [15:23:04] 10VPS-project-codesearch, 10TechCom: Automatically index extensions in CodeSearch - https://phabricator.wikimedia.org/T268328 (10daniel) >>! In T268328#6636807, @Ladsgroup wrote: > Codesearch searches for extensions and rebuild it on daily basis: https://gerrit.wikimedia.org/g/labs/codesearch/+/refs/changes/47... [15:23:45] That's neat indeed :-) [15:27:55] meanwhile I have learned a few things about gerrit [15:27:57] and our setup [15:28:09] such that our gerrit.war actually being releases.war which contains gerrit + the core plugins [15:28:24] but the bundled plugins have to be installed to ./plugins [15:28:25] java -jar release.war init --batch --install-all-plugins [15:28:32] it is not automagic :\ [15:28:59] the bazel build does provide a bundle of all the core plugins as a core.zip but it is not recognized when put inside ./plugins/ (invalid jar) [15:29:20] so one can unzip it then copy the .jar to ./plugins [15:29:34] but really init --install-all-plugins looks easier probably ;] [15:30:01] until I found out that we rely on an unmerged gitiles plugin hack [15:35:03] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10User-DannyS712: Beta cluster logstash down - https://phabricator.wikimedia.org/T268200 (10herron) 05Open→03Resolved a:03herron Apache2 on `deployment-logstash03` was erroring with `[auth_cas:error] [pid 18928:tid 1397677191127... [15:36:27] 10Scap: Scap .deb build patches bin/scap for hashbang - https://phabricator.wikimedia.org/T268349 (10LarsWirzenius) [15:38:16] 10Gerrit, 10Wikimedia-GitHub, 10Patch-For-Review: mediawiki/extensions/WSOAuth Github and Gerrit repo have diverged - https://phabricator.wikimedia.org/T263955 (10Aklapper) @Xxmarijnw: Hi, see the jenkins-bot output on that Gerrit patch about which additional fixes are needed. [15:40:10] Honestly, I wound not use `--install-all-plugins`. We (currently) pull our plugins from our archiva. `--install-all-plugins` only gets in the way there. [15:40:36] hashar: ^ [15:40:50] yes [15:40:53] Did you update with `--install-all-plugins`? [15:41:09] on my local machine yes [15:41:14] Ah! [15:41:28] and confirmed they were the same as from ./bazel-bin/plugins/$PLUGIN/$PLUGIN.jar [15:41:51] it was just easier for me to use --install-all-plugins to refresh the plugin that will eventually land in archiva [15:41:58] rather than running a few cp [15:42:05] Mhmm. Ok. [15:42:50] end result is the same at least [15:43:03] but that made it slightly nicer to my workload yesterday [15:43:15] I rebuild the whole things a few time [15:44:10] Not sure I fully understood everything, but if it works, it works :-D \o/ [15:55:09] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Developer-Advocacy, 10Wikimedia-Tech-Talks: Tech Talks Proposal: new CI system candidate demonstration - https://phabricator.wikimedia.org/T220695 (10srodlund) Great! If this talk does go forward, I propose we move it to... [16:01:56] qchris: yes pretty much :] [16:02:35] (03PS3) 10Jforrester: layout: [labs/tools/book2scroll] Provide CI with tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/642097 (https://phabricator.wikimedia.org/T267488) (owner: 10Jay (CIS-A2K)) [16:03:33] (03CR) 10Jforrester: [C: 03+2] layout: [labs/tools/book2scroll] Provide CI with tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/642097 (https://phabricator.wikimedia.org/T267488) (owner: 10Jay (CIS-A2K)) [16:05:16] (03Merged) 10jenkins-bot: layout: [labs/tools/book2scroll] Provide CI with tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/642097 (https://phabricator.wikimedia.org/T267488) (owner: 10Jay (CIS-A2K)) [16:06:07] !log Zuul: [labs/tools/book2scroll] Provide CI with tox-docker T267488 [16:06:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:06:10] T267488: Rewrite book2scroll tool and deploy on toolforge - https://phabricator.wikimedia.org/T267488 [16:12:12] kostajh brennen and anyone else I should be asking about this... what is the procedure for getting credentials for uploading Docker images to docker-registry.wikimedia.org (for dev images, made from the dev-images repository)? [16:12:14] Thanks in advance! [16:13:46] (03PS1) 10Jforrester: layout: [mediawiki/services/graphoid/deploy] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/642459 (https://phabricator.wikimedia.org/T242855) [16:13:48] (03PS1) 10Jforrester: jjb: Drop graphoid-deploy-npm-node-6-docker, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/642460 [16:13:50] (03PS1) 10Jforrester: dockerfiles: Drop npm-test-graphoid, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/642461 [16:15:48] AndyRussG: It's an SRE priv (defined in puppet). [16:16:36] James_F: ohhhh hmmm [16:16:54] https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Access_list [16:17:40] Can't remember which one it is. [16:17:46] Helpful. ;-) [16:17:57] James_F: that sounds slightly less than ideal for what we've been thinking of doing... Basically since the whole FR-Tech payments/donor management stack is pretty unique, we've been working on our own little Docker setup, but in a corner of the dev-images repo [16:18:24] on the assumption that these will be images that mainly fr-tech folks will care about, work on, update, etc. [16:18:33] James_F: thanks that's super helpful though!!! [16:18:56] fine starting point for figuring out how to proceed [16:19:17] Oh, right, if you want to start minting your own base images for prod use SRE will definitely want to have words. :-) [16:19:18] I guess we could ask FR SREs just to do it [16:19:27] James_F: no not for prod [16:19:30] And you'll want them in your own namespace. [16:19:33] only for developers' local setup [16:19:44] Ah, huh. [16:19:51] i.e. finally moving away from Vagrant spaghetti [16:19:54] Would this be used by CI eventually? [16:20:03] If so, the releng/dev space would make sense, yes. [16:20:07] "eventually". [16:21:11] James_F: hmmm dunno about CI, there's already FR docker stuff happening there [16:21:23] so my guess is no, they wouldn't be used there [16:21:48] Is this for things that aren't covered by CI? Just trying to understand why those images aren't sufficient. [16:23:41] James_F: it's a totally different thing. It's for a unified, standardized setup for fr-tech devs to use on their local machines [16:23:50] Ah, right. [16:24:03] There's the local-dev area which sounds like a good fit, yeah. [16:24:28] we could also use a separate image repo or even a separate dev image configuration repo, but I was kinda hoping for less siloing, not more [16:24:31] AndyRussG, I don't have any context, so I ask the silly question: how long does it take to build the images? is it enough for each dev to just build them themselves? is uploading them to a registry needed? [16:25:33] Oh, yes, a shared dockerfile would be easy to dump in a dev repo, that's true. [16:25:59] liw: it's pretty quick but it's a pain. yes so far people have been building their own images [16:30:30] (03PS1) 10Ahmon Dancy: add rebuild-vm command [tools/train-dev] - 10https://gerrit.wikimedia.org/r/642465 [16:31:38] AndyRussG: we've batted around the idea of having things in dev-images deploy automatically, once merged [16:32:02] for the moment though, how often do you anticipate changes? [16:34:07] (03CR) 10Lars Wirzenius: [C: 03+2] add rebuild-vm command [tools/train-dev] - 10https://gerrit.wikimedia.org/r/642465 (owner: 10Ahmon Dancy) [16:34:14] (03Merged) 10jenkins-bot: add rebuild-vm command [tools/train-dev] - 10https://gerrit.wikimedia.org/r/642465 (owner: 10Ahmon Dancy) [16:35:39] hashar hmm https://gerrit.wikimedia.org/r/642464 doesn’t seem to redirect [16:35:57] I thought we fixedthat yesterday [16:36:02] brennen: hmmm I guess it's an unknown right now, probably maybe initially 2-3 times a month, and then when things settle down, a lot less, like once every 2-3 months? just a wild guess [16:36:23] paladox: worked for me :] [16:37:38] Is the change hidden? [16:37:40] wonder if that one is due to permissions. It does redirect for me logged in [16:38:05] I think it's due to removing READ from refs/* on All-Projects [16:38:21] ah yeah [16:38:21] so [16:38:28] that is a change against refs/meta/config [16:38:29] do releng own releases.wikimedia.org? https://phabricator.wikimedia.org/T268345 [16:38:40] and they are not readable right now [16:38:56] addshore: Yes. [16:39:01] unless you are and Administrator or a Project Owner of the All-Projects.git repository [16:39:07] James_F: cool! [16:39:08] Doesn’t seem to redirect for me when logged in either [16:39:09] Oh [16:39:10] Yeh [16:39:11] Just realised that [16:39:59] brennen liw James_F aaaah gotta be afk for about 20 minutes, thanks so so much for the help and input!!! (I'll also see backscrool here when I'm back) [16:40:03] and in a default Gerrit installation refs/meta/config is not readable [16:44:32] 10phan-taint-check-plugin: Taint data for Sanitizer::removeHTMLtags is incorrect - https://phabricator.wikimedia.org/T268353 (10Daimona) [16:46:19] 10Gerrit, 10Release-Engineering-Team (Development services), 10Regression, 10Upstream: URLs in format https://gerrit.wikimedia.org/r/123456 end up on port 80, leading to error SSL_ERROR_RX_RECORD_TOO_LONG - https://phabricator.wikimedia.org/T268260 (10thcipriani) [16:46:21] 10Gerrit, 10Release-Engineering-Team (Development services): Gerrit: remote unpack failed: Missing tree / blob - https://phabricator.wikimedia.org/T268119 (10thcipriani) [16:46:23] 10Gerrit, 10Release-Engineering-Team: Dashboards disappeared from gerrit - https://phabricator.wikimedia.org/T268053 (10thcipriani) [16:46:59] 10Gerrit, 10Security-Team, 10Patch-For-Review, 10Security: Gerrit security advisory 2020-11-17 - https://phabricator.wikimedia.org/T268009 (10thcipriani) 05Open→03Resolved >>! In T268009#6637118, @gerritbot wrote: > Change 642464 had a related patch set uploaded (by Hashar; owner: Hashar): > [All-Proje... [16:48:24] hashar: revert applied, and graphs looking correct: https://grafana.wikimedia.org/d/yxJKOJTMk/permission?orgId=1&from=now-15m&to=now [16:48:41] nice thans ! [16:48:46] thank you [16:49:59] if my theory is right, thread spikes should also stop: https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=activeThreads [16:50:10] which I will be happy about since gc runs tonight :) [16:50:19] :) [16:51:07] speaking of that [16:51:22] one repo takes like 4 hours to pass through gc [16:51:26] ores/editquality iirc [16:51:44] 10VPS-project-codesearch, 10TechCom: Automatically index extensions in CodeSearch - https://phabricator.wikimedia.org/T268328 (10Krinkle) >>! In T268328#6636960, @daniel wrote: > […] and then has a hard-coded list of additional stuff to index. I'm proposing to change this script so it scans the relevant catego... [16:52:36] and also we have the non default: modules/gerrit/templates/gerrit.config.erb: deltacompression = true [16:52:36] indeed [16:52:43] which might be harming us [16:52:59] I hadn't looked into that setting before [16:53:19] it was mentioned during one of the presentation I attended this week [16:53:25] nice :) [16:53:34] the default is false [16:53:38] no idea what it changes for us [16:54:48] 10VPS-project-codesearch, 10TechCom: Automatically index extensions in CodeSearch - https://phabricator.wikimedia.org/T268328 (10Jdforrester-WMF) What's the problem this is trying to solve? [16:55:05] probably worth testing [16:55:26] AndyRussG: so publishing to the dev/ namespace from dev-images seems like the appropriate thing to me for your use case. i don't know that i see any reason one or more appropriate persons from fr-tech shouldn't be able to log in to contint and run the same deploy job we use for that, but in the meanwhile i would be happy to run deploy jobs for you. [16:55:52] i think i also probably owe you some code review on your changes there. [16:56:24] all the other repos gc in minutes, even mw/core and ops/puppet take less than 10min usually, but that one repo seems to take forever. There don't seem to be large objects in it, just a lot of medium sized objects I guess. [16:57:29] maybe gc can be tuned specially for that repo [16:57:46] 10VPS-project-codesearch, 10TechCom: Automatically index extensions in CodeSearch - https://phabricator.wikimedia.org/T268328 (10Jdforrester-WMF) (Also we shouldn't embed a specific tool like CodeSearch in a long-term policy if it might be scrapped within a year or so – {T268196}.) [16:57:56] its material should probably be offloaded to git fat or git lfs anyway [16:59:14] I dunno. I remember running git-sizer on it at one point and it didn't find anything too amiss there [16:59:40] https://github.com/github/git-sizer [16:59:58] 10VPS-project-codesearch, 10GitLab: Figure out the future of codesearch in a GitLab world - https://phabricator.wikimedia.org/T268196 (10mmodell) fwiw, I believe that gitlab "CE" has a very limited search feature. We might well still need another solution for decent code search. [17:00:16] ahah [17:00:24] a linter for git repositories ... that is meta! [17:01:27] https://phabricator.wikimedia.org/T237807#5651875 [17:01:32] is the report for the one [17:01:47] 7GB in objects :( [17:02:39] that accounts for about 1/6th of all gerrit storage [17:02:47] obese [17:03:37] and yeah, now that I'm re-reading this report, it seems like we could move some things there to git-lfs [17:03:43] 35MB objects [17:06:14] Delete the repo and start again or you'll never get true savings. [17:06:58] you could rebase it to remove most history and then gc --agressive the repo? [17:07:03] for some pain [17:07:35] Well, yeah, if you filter-branch it to hell and back. [17:07:37] But that's cheating. [17:07:49] (We did that to excise Parsoid's git history from VE back in the day.) [17:08:59] yeah, lfs + filter-branch to hell and back was about what I was expecting :) [17:09:55] Deleting the repo saves so much time. [17:10:53] 10VPS-project-codesearch, 10Operations: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199 (10Dzahn) Yes, you are right it would need a security review, that's a good point. That might take a bit to get scheduled. Using webproxy should not be an issue, we do that for other things as... [17:13:49] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10brennen) Yeah, I know that we [[https://wikitech.wikimedia.org/wiki/Production_access#Security | disallow agent forwarding]], at least by policy and... [17:13:50] I do love the feeling of deleting code :) [17:17:03] 10phan-taint-check-plugin, 10Patch-For-Review: Taint data for Sanitizer::removeHTMLtags is incorrect - https://phabricator.wikimedia.org/T268353 (10Daimona) a:03Daimona [17:19:59] 10VPS-project-codesearch, 10GitLab: Figure out the future of codesearch in a GitLab world - https://phabricator.wikimedia.org/T268196 (10hashar) Gitlab CE comes with a search system backed up by the database ( https://docs.gitlab.com/ee/user/search/#basic-search ). Searches for commit, code or comments are lim... [17:21:55] thcipriani: liw: James_F: for scoring/ores/editquality , we probably don't need the history anyway [17:22:04] could be checked with the product owners [17:22:20] hm, in that case, why is it a git repository? [17:22:54] well [17:23:00] I would rather not answer :] [17:23:12] more seriously I think the need was to be able to rollback a model [17:23:17] and git sounds natural for that [17:23:27] and hosted on gerrit for deployment purpose [17:23:38] all that most probably pre date Archiva / LFS / git FAT [17:23:47] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [17:23:49] I think they were the first to adopt git lfs to offload gerrit [17:23:55] and we later went with git-fat for other projects [17:23:59] Yeah [17:24:05] 10phan-taint-check-plugin: Caused-by lines apparently picking up everything that is concatenated - https://phabricator.wikimedia.org/T250679 (10Daimona) 05Open→03Resolved [17:24:08] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [17:24:08] Reasonable decisions at the time. [17:24:08] so it might just be some left over and we can probably archive that repo out of gerrit view [17:24:12] 10phan-taint-check-plugin: Caused-by lines include lines for unrelated types of taintedness - https://phabricator.wikimedia.org/T257351 (10Daimona) 05Open→03Resolved [17:24:17] or just plain delete it if it serves no purposes nowadays [17:24:46] https://phabricator.wikimedia.org/T238746 is the generic task to cleanup their git repos [17:24:59] maybe we can raise it to them [17:25:45] anyway, i had long days, see you next week [17:25:47] or at least run gc on it rarely? [17:25:54] will see what we do with the train on monday ;] [17:26:11] 10Gerrit, 10Release-Engineering-Team (Development services), 10Analytics: Unable to clone git repo from stat1008 - https://phabricator.wikimedia.org/T268290 (10Dzahn) >>! In T268290#6636347, @kostajh wrote: > Weirdly (to me anyway) there is no SSH key visible in my home directory on the server SSH keys are... [17:26:13] 10phan-taint-check-plugin: Taint data for Sanitizer::removeHTMLtags is incorrect - https://phabricator.wikimedia.org/T268353 (10Daimona) 05Open→03Resolved [17:26:14] sleep well, hashar; be well, be safe, be happy [17:31:20] brennen: ahh cool thanks!!! [17:32:02] brennen: also thanks in advance for any suggestions on the image setup... we were actually talking just yesterday about making some further tweaks on the fundraising dev apache image ;p [17:38:04] 10phan-taint-check-plugin: Expand taintedness data for builtin functions - https://phabricator.wikimedia.org/T256660 (10Daimona) 05Open→03Resolved [17:46:37] liw :))) [17:46:41] bbl [17:52:32] Daimona: https://gerrit.wikimedia.org/r/q/project:mediawiki/tools/phan/SecurityCheckPlugin+status:open+is:mergeable is somewhat emptier. :_) [17:54:08] 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), 10Release, 10Train Deployments: 1.36.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T263186 (10Umherirrender) [17:54:48] James_F: Eheh it really is :D Thank you! [17:55:35] Daimona: Thank *you*. It's great how much better we are at finding and fixing these bugs before they break the world now, and it's so much due to phan and taint-check especially. [17:56:58] 10phan-taint-check-plugin: Make taint-check stronger about abnormal syntax by using test files from codesniffer - https://phabricator.wikimedia.org/T251029 (10Daimona) 05Open→03Resolved [17:57:29] Well, I'm also happy about that. Also, the new release seems to be great, lots of "unused suppression", almost 0 false positives from taint-check, and some true positives. [17:57:43] Yeah. [17:57:56] The unused ternary one in ContentTranslation made me sad. [17:58:04] I mean, great that we found it, but… [17:58:15] I owe you a lot for the CR! [17:58:30] Happy to help. [17:58:33] Uh, yes, that was a horrible bug... [18:00:32] As for taint-check, I guess it's almost out of beta [18:00:36] 10phan-taint-check-plugin: Uncaught throwable for "dynamic" hooks in taint-check 3 - https://phabricator.wikimedia.org/T250371 (10Daimona) 05Open→03Resolved [18:00:41] Almost. ;-) [18:00:53] Yeah some big patches are still needed [18:01:07] One day I should try the pass-by-ref one, but I get lost every time I look at it [18:01:18] Partly because the implementation upstream is (was?) incomplete [18:01:47] Fun. [18:10:56] 10phan-taint-check-plugin: Foreach loops hide the "outer" source of taintedness in caused-by lines - https://phabricator.wikimedia.org/T257348 (10Daimona) 05Open→03Resolved [18:10:58] 10phan-taint-check-plugin, 10Patch-For-Review: Fix caused-by lines in phan-taint-check - https://phabricator.wikimedia.org/T203652 (10Daimona) [19:09:16] 10Release-Engineering-Team, 10MediaWiki-skins-Daddio: Sunset Daddio skin - https://phabricator.wikimedia.org/T267455 (10Jdlrobson) [19:09:32] 10Release-Engineering-Team, 10MediaWiki-skins-Daddio, 10Technical-Debt (Deprecation process): Sunset Daddio skin - https://phabricator.wikimedia.org/T267455 (10Jdlrobson) [19:26:21] 10VPS-project-codesearch, 10TechCom: Automatically index extensions in CodeSearch - https://phabricator.wikimedia.org/T268328 (10Legoktm) In addition to what James and Krinkle said, I would also add that the goal of codesearch is not to index any MediaWiki code ever written that was dumped into a git repo. If... [20:19:09] (03PS4) 10Kosta Harlan: Linkrecommendation service example [releng/local-charts] - 10https://gerrit.wikimedia.org/r/639347 (owner: 10Jeena Huneidi) [20:23:47] 10Gerrit, 10Wikimedia-GitHub, 10Patch-For-Review: mediawiki/extensions/WSOAuth Github and Gerrit repo have diverged - https://phabricator.wikimedia.org/T263955 (10Xxmarijnw) @Aklapper I have uploaded a new patchset that fixes the error generated by PHPUnit. The current build is still failing because [[ htt... [22:16:25] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Summarize experiments with buildkit based MediaWiki image builds - https://phabricator.wikimedia.org/T268368 (10dduvall) [22:16:49] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)): Summarize experiments with buildkit based MediaWiki image builds - https://phabricator.wikimedia.org/T268368 (10dduvall) p:05Triage→03Medium [23:45:17] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [23:47:56] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [23:49:36] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10serviceops, 10Patch-For-Review: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn)