[00:01:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Please install 'npx' node package runner on CI nodes - https://phabricator.wikimedia.org/T246400 (10Krinkle) I suggest declining as this seems to be due to a misunderstanding. See [my CR comment](https://gerrit.wikimedia.org/r/#/c/mediaw... [00:16:08] 10Deployments, 10Parsing-Team, 10serviceops: Add a parse* host to the scap canary checks? - https://phabricator.wikimedia.org/T246728 (10Jdforrester-WMF) [00:18:07] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap, 10Wikimedia-Incident: Make scap skip restarting php-fpm when using --force - https://phabricator.wikimedia.org/T243009 (10Krinkle) Does the "maybe restart php-fpm" step take a noticable amou... [01:07:22] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap, 10Wikimedia-Incident: Make scap skip restarting php-fpm when using --force - https://phabricator.wikimedia.org/T243009 (10thcipriani) >>! In T243009#5935322, @Krinkle wrote: > Does the "mayb... [01:10:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Please install 'npx' node package runner on CI nodes - https://phabricator.wikimedia.org/T246400 (10thcipriani) 05Open→03Declined >>! In T246400#5935289, @Krinkle wrote: > I suggest declining as this seems to be due to a misunderstan... [03:25:51] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10Aklapper) @DStrine: The problem that I pointed out in my last comment is not resolved. Do you want me to create a separate task for that? [05:00:00] 10Project-Admins, 10PM: Mass-migrate project tags for parsing team - https://phabricator.wikimedia.org/T245868 (10Aklapper) a:05Aklapper→03ssastry Reassigning to @ssastry for the two remaining items in T245868#5929537 [05:07:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Please install 'npx' node package runner on CI nodes - https://phabricator.wikimedia.org/T246400 (10Demian) npm scripts handle calling local *and* global module binaries, but npx also handles installing the module temporarily, if not ins... [06:33:44] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Developer-Advocacy: Tech Talks Proposal: new CI system candidate demonstration - https://phabricator.wikimedia.org/T220695 (10Aklapper) [06:35:26] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Developer-Advocacy: Tech Talks Proposal: new CI system candidate demonstration - https://phabricator.wikimedia.org/T220695 (10Aklapper) [06:51:49] 10Beta-Cluster-Infrastructure, 10JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779 (10ArielGlenn) Wikidata alone has 78 million main namespace pages. You'd be talking about a huge leap in the amount of storage required. I don't know what thought... [06:58:05] 10Project-Admins, 10Community-Tech: Create a Phabricator project tag for "Commons deletion notification bot" - https://phabricator.wikimedia.org/T229759 (10Samwilson) A [[https://phabricator.wikimedia.org/search/query/9LccxtrPr1EB/#R | search]] turns up quite a few, many of which would probably belong in a pro... [07:16:22] Project beta-code-update-eqiad build #286532: 04FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/286532/ [07:24:54] Yippee, build fixed! [07:24:54] Project beta-code-update-eqiad build #286533: 09FIXED in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/286533/ [07:30:09] 10Gerrit, 10Analytics, 10Gerrit-Privilege-Requests, 10User-MarcoAurelio: Give access to Wikistats 2 to l10n-bot - https://phabricator.wikimedia.org/T245805 (10abi_) @MarcoAurelio - Currently patches submitted by the l10n-bot (https://gerrit.wikimedia.org/r/c/analytics/wikistats2/+/576042) still require a m... [07:41:21] 10Gerrit, 10Analytics, 10Gerrit-Privilege-Requests, 10User-MarcoAurelio: Give access to Wikistats 2 to l10n-bot - https://phabricator.wikimedia.org/T245805 (10fdans) @abi_ thank you for all the help with this. I'm not sure how it works in other projects, but manually merging every patch seems prone to prob... [08:04:14] 10Gerrit, 10Analytics, 10Gerrit-Privilege-Requests, 10User-MarcoAurelio: Give access to Wikistats 2 to l10n-bot - https://phabricator.wikimedia.org/T245805 (10abi_) > If so we can close this task. To clarify, I meant {T240621} >>! In T245805#5935833, @fdans wrote: > @abi_ thank you for all the help with... [08:09:30] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: BlueSpice related tests fails on gate-and-submit-1.31 in quibble-composer tests - https://phabricator.wikimedia.org/T235807 (10Osnard) I am sorry, but I think we might have mixed things up in this discuss... [08:38:58] (03PS1) 10Thiemo Kreuz (WMDE): Report particularly confusing assertNotEquals( false ) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/576276 [09:49:17] 10Release-Engineering-Team (Code Health), 10Code-Stewardship-Reviews, 10MediaWiki-extensions-FlaggedRevs: Code stewardship review: FlaggedRevs - https://phabricator.wikimedia.org/T185664 (10Zache) Bug: Try to edit a specific page on deWP throws InvalidArgumentException T246720 [10:09:40] 10Beta-Cluster-Infrastructure: Fatal error: Class 'LoggedUpdateMaintenance' not found when using mwscript - https://phabricator.wikimedia.org/T246754 (10Daimona) [10:12:09] 10Beta-Cluster-Infrastructure: deployment-mediawiki-07 socket timeout 2020-03-01/02 - https://phabricator.wikimedia.org/T246577 (10AlexisJazz) Unable to modify the file "76f2dbef4cc9_7.jpg" because the file repository "local" is in read-only mode. The system administrator who locked it offered this explanation:... [10:50:35] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10Developer Productivity, and 3 others: Command-line wrapper for interacting with core's docker-compose stack - https://phabricator.wikimedia.org/T246111 (10kostajh) > @kostajh: I'd like to collaborat... [11:08:07] parsoid is broken on en.wikipedia.beta; I didn't see a task for this about phab but if it's not known I can create one. [11:10:43] 10Beta-Cluster-Infrastructure: Parsoid integration is broken on beta cluster sites - https://phabricator.wikimedia.org/T246760 (10kostajh) [11:11:38] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Quality-and-Test-Engineering-Team (QTE): Parsoid integration is broken on beta cluster sites - https://phabricator.wikimedia.org/T246760 (10kostajh) [11:11:53] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Quality-and-Test-Engineering-Team (QTE): Parsoid integration is broken on beta cluster sites - https://phabricator.wikimedia.org/T246760 (10kostajh) [12:01:37] 10Gerrit, 10Wikimedia-production-error (Shared Build Failure): 502 errors when cloning repos for wmf-quibble-vendor-mysql-php72-docker and mwext-php72-phan-docker jobs - https://phabricator.wikimedia.org/T246763 (10kostajh) [12:13:56] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T233870 (10LarsWirzenius) Started cutting the branch for this week's train. [12:19:52] 10Gerrit, 10Wikimedia-production-error (Shared Build Failure): 502 errors when cloning repos for wmf-quibble-vendor-mysql-php72-docker and mwext-php72-phan-docker jobs - https://phabricator.wikimedia.org/T246763 (10Lucas_Werkmeister_WMDE) This has been happening occasionally for a while now, but it does seem t... [13:24:15] (03PS1) 10Kosta Harlan: (wip) Add ElasticSearch dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) [14:32:40] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Fatal error: Class 'LoggedUpdateMaintenance' not found when using mwscript - https://phabricator.wikimedia.org/T246754 (10Daimona) 05Open→03Resolved a:03Daimona [14:45:55] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10DStrine) Sure but as far as I can tell there isn't anything that can be done about it. [15:23:14] !log mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=eswikibooks | T246539 [15:23:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:23:16] T246539: Dry-run, then actually run updateVarDumps - https://phabricator.wikimedia.org/T246539 [15:35:39] hey scappers, didn't scap have an option to just sync to a single host give at command line? [15:36:00] so not all servers but also not having to run 'scap pull' on that on eserver directly [16:02:16] mutante: hrm, I don't remember it having that option. You could likely override the dsh_targets option with a list of 1 server if that's useful, otherwise scap pull is all I remember. [16:04:55] thcipriani: thanks! hmm. There is an expectation that a users who are both in 'deployment' and also parsoid-admin (users who can ssh to wtp* and restart the service) can also run "scap pull" on just a single server. so one option would have been to run stuff as deployment on deployment* and the other to get sudo privileges for 'scap pull' locally. [16:05:13] and then additionally i can 'scap pull' without ever typing sudo of course [16:05:46] but if i become a non-ops user, the this happens: [16:05:48] 15:44:55 Started rsync common [16:05:49] sudo: a password is required [16:05:49] 15:44:55 Finished rsync common (duration: 00m 00s) [16:06:03] so now i am trying to see the sudo lines within scap etc [16:08:03] being able to say something like "just ran scap --host wtp1025 .. on deploy1001" would have been nice though [16:10:55] i think i'll add "scap pull as user mwdeploy" to sudo privileges of parsoid-admin then [16:15:30] (03PS1) 10Gabrielchihonglee: Add gabrielchl/gabrielchihonglee to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/576382 [16:18:51] mutante: hrm...this is tricky. Internally scap runs a few commands as the mwdeploy user and the l10nuser -- rsync and an mwscript command. IIRC, the deployers group is able to run commands as those users. Adding parsoid folks to the deploy group may be easier? Dunno. [16:21:51] thcipriani: Yea, i see saw those lines for running things as mwdeploy and l10nuser. Though, the deployment group is not on wtp* hosts, just on deployment_hosts. So what i would do is add a subset of that to the parsoid-admins group which is the local group on wtp* [16:23:07] probaly only specifically "scap pull as mwdeploy" [16:29:05] yea, tested it. https://gerrit.wikimedia.org/r/c/operations/puppet/+/576383/1/modules/admin/data/data.yaml works with "sudo -u mwdeploy scap pull". the l10nuser part is not needed [16:29:56] (on wtp1025 as cscott) [16:32:40] 10Release-Engineering-Team-TODO, 10Scap, 10Regression: Scap no longer sets 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10Krinkle) [16:37:52] 10Gerrit, 10Analytics, 10Gerrit-Privilege-Requests, 10User-MarcoAurelio: Give access to Wikistats 2 to l10n-bot - https://phabricator.wikimedia.org/T245805 (10MarcoAurelio) >>! In T245805#5935811, @abi_ wrote: > @MarcoAurelio - Currently patches submitted by the l10n-bot (https://gerrit.wikimedia.org/r/c/a... [16:40:12] 10Release-Engineering-Team-TODO, 10Scap, 10User-brennen, 10Wikimedia-Incident: Add a quick rollback command to scap that skips all unnecessary steps - https://phabricator.wikimedia.org/T244544 (10brennen) [16:44:19] 10Gerrit, 10Analytics, 10Gerrit-Privilege-Requests, 10Patch-For-Review, 10User-MarcoAurelio: Give access to Wikistats 2 to l10n-bot - https://phabricator.wikimedia.org/T245805 (10MarcoAurelio) 05Resolved→03Open Reopening due to new patch for review. [16:53:15] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Samantha_Alipio_WMDE) Hi, could you please add me to the #acl_project-admins group? I am a PM with WMDE and will be creating projects on an ongoing basis. [17:05:45] 10Phabricator, 10Operations, 10Security-Team, 10Security: Adjust onboarding/offboarding logic to accommodate changes to #security (now acl*security) - https://phabricator.wikimedia.org/T245771 (10jbond) 05Resolved→03Open reopen this ticket as we need to ensure it handles the subgroups [17:51:05] Hello there! A question: what's the best way to run a write query on the beta cluster? T246539 screwed up a couple of lines [17:51:06] T246539: Dry-run, then actually run updateVarDumps - https://phabricator.wikimedia.org/T246539 [17:51:50] What do you mean by write a query? [17:51:54] Run a select? [17:52:01] No, a "write query" [17:52:10] Update stuff [17:52:15] you can do `sql --write dbname` [17:52:22] Gotcha, thanks [17:52:25] or `mwscript sql.php dbname` [17:52:32] And log it here, right? [17:53:02] probably not a bad idea [17:53:15] Hmmm [17:53:39] Actually, reverting the changes might be too complicated for pure SQL. Is it possible to restore a backup or sth like that? [17:53:40] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10ChangeProp, 10Release Pipeline, and 3 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10hnowlan) [17:53:55] Or just leave it broken, but that seems sad. [17:54:53] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10ChangeProp, 10Release Pipeline, and 3 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10hnowlan) changeprop now deploys to staging and creates pods successfully. Errors out with this, as expected... [17:58:05] AFAIK we don't have backups at all for beta [18:02:02] Alright [18:02:12] I'll try something [18:06:01] 10Release-Engineering-Team-TODO, 10Wikimedia-Portals: Update the wikimedia-portals repo's CI/linting code for various security issues - https://phabricator.wikimedia.org/T237479 (10Jdforrester-WMF) [18:06:03] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Wikimedia-Portals, 10serviceops: Migrate www.wikimedia.org (the portal) to be hosted as a service - https://phabricator.wikimedia.org/T238747 (10Jdforrester-WMF) [18:11:45] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T233870 (10LarsWirzenius) Group0 deployed earlier today, and looking OK so far. [18:17:24] 10Release-Engineering-Team-TODO, 10Wikimedia-Portals: Update the wikimedia-portals repo's CI/linting code for various security issues - https://phabricator.wikimedia.org/T237479 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Well, bother. I was about to mark this as Resolved, but there's a new... [18:17:26] 10Continuous-Integration-Infrastructure (Slipway), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Release Pipeline, 10Wikimedia-Portals: Migrate wikimedia-portals-build to Docker container - https://phabricator.wikimedia.org/T213806 (10Jdforrester-WMF) [18:17:57] portals needed some love James_F - thanks for helping :) [18:22:11] !log Manually fixing eswikibooks DB rows with P10600 refs T246539 [18:22:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:22:13] T246539: Dry-run, then actually run updateVarDumps - https://phabricator.wikimedia.org/T246539 [18:27:47] hauskatze: I'd dearly love to move them to k8s, but it's waiting on SRE creating a base image. :-( [18:28:09] (And they've got more urgent things to work on.) [18:28:25] ugh, k8s [18:28:29] complex for me [18:28:40] Psh. One blubber file and you're away. [18:28:48] Never look at integration/config again! Etc. [18:29:02] I am yet to migrate my Toolforge cronjobs from crontab to kubernetes [18:29:11] Tut. [18:29:25] the config file scared the hell out of me [18:29:27] heh [18:29:51] I find layout.yaml some of the most worrying code we have, TBH. [18:30:08] One typo and suddenly we don't test a vital bit of infrastructure, etc. [18:46:23] hauskatze: very few folks have made that switch in Toolforge. You are 100% correct that directly creating the YAML to make cronjobs there is complicated. We are going to think about how to make that all a bit easier for folks in the future. Grid cronjobs will be with us for quite a while still. [18:52:18] James_F: would it be possible to backport https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/576405/ ? [18:52:21] bd808: I see. I'm not blaming anyone but myself just to be clear. [18:54:37] hauskatze: I just checked and there is exactly 1 tool using Kubernetes CronJob objects today. You are not behind the curve here at all. :) [18:56:49] Daimona: Were you planning to run it in prod this week? [18:57:27] Daimona: I mean, yes, if you wish, just surprised. :-) [18:59:50] It depends... The sooner the better, I'd say [19:00:05] But it can wait, of course [19:00:23] (As long as it's ready before the 1.35 cut) [19:03:04] Yeah, I'm a little nervous about the churn. [19:11:38] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 2 others: Include WikibaseLexemeCirrusSearch into CirrusSearch tests - https://phabricator.wikimedia.org/T244485 (10EBernhardson) p:05Triage→03Medium [19:15:04] No prob, thanks! [19:27:19] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10Aklapper) @DStrine: Sure something can be done: Allowing a second person to edit #acl_wmf-fr, so there is a fallback in place. @mepps looks like an obvious candidate to me? [19:29:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Please install 'npx' node package runner on CI nodes - https://phabricator.wikimedia.org/T246400 (10Demian) @Krinkle follow-up to your [[ https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MultimediaViewer/+/574636#message-c81920e957c... [19:32:14] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10MarcoAurelio) Maybe also allow @offboarding (the account the [[ https://github.com/wikimedia/puppet/blob/production/modules/openldap/files/offboard-user.py | offboard-user.py scri... [19:37:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Please install 'npx' node package runner on CI nodes - https://phabricator.wikimedia.org/T246400 (10Krinkle) @Demian It is not about making it explicit/clear. The approach of installing and executing such code without version lock is not... [19:46:09] 10Release-Engineering-Team (Kanban), 10Scap: Automate updating deployment notes - https://phabricator.wikimedia.org/T196516 (10thcipriani) 05Resolved→03Open a:05thcipriani→03None This automation has been broken for a bit now, reopening. [19:46:11] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10Epic, 10Goal: Automate the Train - https://phabricator.wikimedia.org/T196515 (10thcipriani) [20:07:52] 10Release-Engineering-Team (Kanban), 10Scap: Automate updating deployment notes - https://phabricator.wikimedia.org/T196516 (10thcipriani) What we already know: zuul doesn't trigger jobs for this: ` [thcipriani@contint1001 ~]$ zuul enqueue --trigger gerrit --pipeline post --project mediawiki/core --change 576... [20:16:42] 10Release-Engineering-Team (Kanban), 10Scap: Automate updating deployment notes - https://phabricator.wikimedia.org/T196516 (10hashar) Got broken by 925a04a6ada4c39e067e78f45fb1247447f7d5ad / https://gerrit.wikimedia.org/r/#/c/integration/config/+/571849/ (T243330) ` lang=yaml,name=zuul/layout.yaml job: - na... [20:42:22] (03PS1) 10Thcipriani: branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 [20:42:47] (03CR) 10jerkins-bot: [V: 04-1] branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 (owner: 10Thcipriani) [20:44:27] ohhh flake8 [20:44:42] (03PS2) 10Thcipriani: branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 [20:53:00] (03PS3) 10Thcipriani: branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 [20:53:24] (03CR) 10jerkins-bot: [V: 04-1] branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 (owner: 10Thcipriani) [20:54:36] (03PS4) 10Thcipriani: branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 [21:11:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO: Please install 'npx' node package runner on CI nodes - https://phabricator.wikimedia.org/T246400 (10Demian) >>! In T246400#5938546, @Krinkle wrote: > As for why it is missing, no it was not intentionally omitted. We build npm directly fr... [21:16:12] (03CR) 10Jforrester: [C: 03+2] branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 (owner: 10Thcipriani) [21:16:41] (03Merged) 10jenkins-bot: branch.py: update defines regex [tools/release] - 10https://gerrit.wikimedia.org/r/576442 (owner: 10Thcipriani) [21:41:12] (03PS2) 10Kosta Harlan: (wip) Add ElasticSearch dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) [21:57:05] (03CR) 10Thcipriani: [C: 03+2] change train-deploy-notes branch regex and files [integration/config] - 10https://gerrit.wikimedia.org/r/575575 (owner: 1020after4) [21:57:58] (03Merged) 10jenkins-bot: change train-deploy-notes branch regex and files [integration/config] - 10https://gerrit.wikimedia.org/r/575575 (owner: 1020after4) [21:59:04] 10Release-Engineering-Team (Kanban), 10Scap: Automate updating deployment notes - https://phabricator.wikimedia.org/T196516 (10thcipriani) 05Open→03Resolved a:03thcipriani >>! In T196516#5938707, @hashar wrote: > I guess this task can be resolved again and we can follow up on T243330 Ah! Lost that menta... [21:59:05] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10Epic, 10Goal: Automate the Train - https://phabricator.wikimedia.org/T196515 (10thcipriani) [22:00:22] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/575575 [22:00:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:06:24] (03PS3) 10Kosta Harlan: (wip) Add ElasticSearch dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) [22:07:50] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: train-deploy-notes Jenkins job fails in conjunction with branch.py - https://phabricator.wikimedia.org/T243330 (10thcipriani) With @mmodell's latest change: https://gerrit.wikimedia.o... [22:09:40] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Parsoid: Parsoid/RESTbase seems to be unavailable in Beta - https://phabricator.wikimedia.org/T246833 (10Ryasmeen) [22:26:36] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Developer Productivity, 10Patch-For-Review: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10Dvorapa) Review needed, Gerrit 2.15 reached EOL nearly 4 months ago! [22:28:30] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: train-deploy-notes Jenkins job fails in conjunction with branch.py - https://phabricator.wikimedia.org/T243330 (10thcipriani) While I can run the clone operation locally on contint1001... [22:32:19] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Developer Productivity, 10Patch-For-Review: Upgrade to Gerrit 2.16.13 - https://phabricator.wikimedia.org/T200739 (10greg) >>! In T200739#5939215, @Dvorapa wrote: > Review needed, Ger... [22:33:46] (03PS2) 10Jforrester: jjb: add cache dir for phan-testrun [integration/config] - 10https://gerrit.wikimedia.org/r/575600 (https://phabricator.wikimedia.org/T226117) (owner: 10Daimona Eaytoy) [22:34:06] 10Gerrit, 10Release-Engineering-Team (Development services), 10Patch-For-Review: Prepare for Polymer 2.x - https://phabricator.wikimedia.org/T227509 (10Dvorapa) [22:41:29] (03PS3) 10Jforrester: jjb: add cache dir for phan-testrun [integration/config] - 10https://gerrit.wikimedia.org/r/575600 (https://phabricator.wikimedia.org/T226117) (owner: 10Daimona Eaytoy) [22:41:33] (03PS1) 10Jforrester: dockerfiles: [mediawiki-phan-testrun] Use composer remove --no-update [integration/config] - 10https://gerrit.wikimedia.org/r/576462 [22:42:07] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [mediawiki-phan-testrun] Use composer remove --no-update [integration/config] - 10https://gerrit.wikimedia.org/r/576462 (owner: 10Jforrester) [22:42:25] (03PS4) 10Jforrester: jjb: add cache dir for phan-testrun [integration/config] - 10https://gerrit.wikimedia.org/r/575600 (https://phabricator.wikimedia.org/T226117) (owner: 10Daimona Eaytoy) [22:43:02] (03Merged) 10jenkins-bot: dockerfiles: [mediawiki-phan-testrun] Use composer remove --no-update [integration/config] - 10https://gerrit.wikimedia.org/r/576462 (owner: 10Jforrester) [22:43:09] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Parsoid: Parsoid/RESTbase seems to be unavailable in Beta - https://phabricator.wikimedia.org/T246833 (10ssastry) @Jdforrester-WMF thinks this is because of yesterday's changes to wmf-config that got merged and deployed where Parsoid is run from vendor/... [22:43:59] !log Docker: Publish mediawiki-phan-testrun 0.1.4 [22:44:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:47:46] (03CR) 10Jforrester: [C: 03+2] "Deployed for the --no-update fix, but this doesn't appear to make a difference." [integration/config] - 10https://gerrit.wikimedia.org/r/575600 (https://phabricator.wikimedia.org/T226117) (owner: 10Daimona Eaytoy) [22:48:38] (03Merged) 10jenkins-bot: jjb: add cache dir for phan-testrun [integration/config] - 10https://gerrit.wikimedia.org/r/575600 (https://phabricator.wikimedia.org/T226117) (owner: 10Daimona Eaytoy) [22:48:46] (03PS2) 10Jforrester: Add gabrielchl/gabrielchihonglee to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/576382 (owner: 10Gabrielchihonglee) [22:50:00] (03CR) 10Jforrester: [C: 03+2] Add gabrielchl/gabrielchihonglee to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/576382 (owner: 10Gabrielchihonglee) [22:50:50] (03Merged) 10jenkins-bot: Add gabrielchl/gabrielchihonglee to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/576382 (owner: 10Gabrielchihonglee) [22:51:25] !log Zuul: Add gabrielchihonglee to CI allow list [22:51:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:04:41] 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Parsoid: Parsoid/RESTbase seems to be unavailable in Beta - https://phabricator.wikimedia.org/T246833 (10Jdforrester-WMF) Yeah, this is probably that the Parsoid server(s?) in Beta Cluster don't have SERVERGROUP set to 'parsoid'. [23:06:20] Krenair: Where is the hieradata for Beta Cluster? [23:13:05] James_F: /puppet/hieradata/cloud/eqiad1/deployment-prep/common.yaml should be it [23:13:11] unless there is more in Horizon too [23:13:15] Found it. Patch just pushed. [23:13:21] mutante: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/576466 [23:14:16] ah, that. ok, merging it [23:14:22] Thanks. [23:14:38] Does it need manually pulling or does that part of Beta Cluster auto-follow puppet still? [23:14:38] (we could use prefix puppet to match all hosts starting mediawiki-parsoid but i also prefer using the repo somehow) [23:14:40] * James_F loses track. [23:14:51] * James_F nods. [23:15:18] i am not sure if manual pull is needed [23:15:49] it's on the prod master now though [23:16:15] nah.. it should just be waiting for a few [23:16:20] OK. [23:20:11] "The last Puppet run was at Tue Mar 3 22:52:34 UTC 2020 (27 minutes ago)." [23:25:18] hi [23:25:26] that'll mean it will run in like 3 minutes probably [23:25:59] though [23:26:16] someone has some uncommitted changes on deployment-puppetmaster [23:27:12] Ah, so it won't apply? [23:27:24] Puppet is murky to me, at bet. [23:27:26] +s [23:28:03] looks like it hasn't got any puppet changes since 21st Feb [23:28:16] That probably doesn't help. [23:28:30] looking at this diff... [23:29:18] https://phabricator.wikimedia.org/P10602 [23:29:51] that restarts.pp change looks suspiciously like it may be part of the reason for those OOM issues we were seeing on deployment-mediawiki hosts? [23:30:16] anyway stashing it all [23:30:39] ugh merge conflits [23:30:55] Fun. [23:31:24] (03CR) 10Jforrester: "This looks fine, but I don't use puppet." [integration/config] - 10https://gerrit.wikimedia.org/r/575012 (owner: 10Jbond) [23:31:30] the last one (mediawiki::webserver) is E.ffie sending apache logs to logstash [23:31:38] it should be good to merge because it only applies "if $stream_to_logstash " [23:31:55] with an old version of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/571239/ [23:32:11] dropped and rebased [23:32:20] no idea about the kafka one. agree about the restarts.pp but don't know the details why it is commented [23:32:29] James_F, alright so puppet agent should pick up new changes now [23:32:37] cool [23:32:59] Excellent, thank you.