[00:16:48] 10Phabricator (Upstream), 10Upstream: Public log for Phabricator user account role change (e.g. deactivation, admin rights) - https://phabricator.wikimedia.org/T102576 (10epriestley) I'm theoretically supportive of this but it's not obvious to me how I'd implement it. Did you have a plan in mind? The approach... [00:23:04] 10Phabricator (Upstream), 10Upstream: Public log for Phabricator user account role change (e.g. deactivation, admin rights) - https://phabricator.wikimedia.org/T102576 (10Krenair) These logs should be fully visible to all users (public, if the instance supports that) and impossible to hide. [02:07:54] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [04:16:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Discovery-Search: quibble-vendor-mysql-hhvm-docker for WikibaseCirrusSearch takes over 40 minutes - https://phabricator.wikimedia.org/T222757 (10Smalyshev) I also think it's probably better to run regular patch set on 7.2 and do hhvm only... [04:50:46] 10Phabricator, 10PHP 7.2 support, 10Patch-For-Review: Move Phabricator from PHP 7.0 to PHP 7.2 - https://phabricator.wikimedia.org/T151070 (10Dzahn) 05Open→03Resolved a:03Dzahn [04:50:48] 10Phabricator, 10PHP 7.2 support, 10Patch-For-Review: Move Phabricator from PHP 7.0 to PHP 7.2 - https://phabricator.wikimedia.org/T151070 (10Dzahn) phab is on phab1003 now which runs PHP 7.2 [04:50:51] 10Phabricator, 10Performance: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T125357 (10Dzahn) [04:50:54] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) 05Stalled→03Open a:03Dzahn [04:51:07] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) [05:06:10] 10Beta-Cluster-Infrastructure: Request to be a interface-admin on zh Beta Cluster - https://phabricator.wikimedia.org/T224204 (10A2093064) [05:33:54] 10Phabricator, 10observability: don't page all of SRE for phabricator 'phd' service not running - https://phabricator.wikimedia.org/T224205 (10Dzahn) [06:17:07] 10Diffusion, 10Repository-Admins: Diffusion: rMW does not update anymore - https://phabricator.wikimedia.org/T224184 (10Mainframe98) 05Open→03Invalid Seems to work now. I assume the move to phab1003 functioned as the restart it needed. [06:47:36] What accounts does https://logstash-beta.wmflabs.org/ use? [07:07:51] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:17:09] 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10hashar) > @Volans wrote: > I'm assuming that in cases in which the rebase fails because of conflicts or the CI fails after the rebase Jenkins would vote -1 and the patch would b... [08:50:44] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) [08:52:05] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) [08:55:28] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) Thanks to @Sbisson / @Reedy and @Jdforrester-WMF , FlaggedRevs faulty changes have been reverted and I have upgraded group 1 to 1.... [09:23:04] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) [09:23:16] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) And I forgot {T224093} [09:43:06] 10Diffusion, 10Repository-Admins: Diffusion: rMW does not update anymore - https://phabricator.wikimedia.org/T224184 (10Mainframe98) See {T151070} for tasks related to the move. [09:48:16] 10Phabricator (Upstream), 10Upstream: Public log for Phabricator user account role change (e.g. deactivation, admin rights) - https://phabricator.wikimedia.org/T102576 (10mmodell) The most common situations I can think of are offboarding and spam/abuse, both of which do seem like they are fine to make public.... [09:52:08] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) [10:06:06] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10Reedy) >>! In T220731#5207403, @hashar wrote: > Thanks to @Sbisson / @Reedy and @Jdforrester-WMF , FlaggedRevs faulty changes have been re... [10:38:28] 10Phabricator, 10observability: don't page all of SRE for phabricator 'phd' service not running - https://phabricator.wikimedia.org/T224205 (10Volans) p:05Triage→03Normal [10:40:44] 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) Seeing https://phabricator.wikimedia.org/T224173 you may also make the template explicitly s... [10:48:19] 10Continuous-Integration-Infrastructure, 10Wikimedia-General-or-Unknown, 10Core Platform Team (Code Health (TEC13)), 10Core Platform Team Kanban (Doing), and 2 others: Deprecate/obsolete $wgWikimediaJenkinsCI - https://phabricator.wikimedia.org/T200650 (10Maintenance_bot) [10:48:54] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 7 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Maintenance_bot) [10:50:59] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10User-Urbanecm, 10User-greg: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10zeljkofilipin) 👍 from me. [10:51:16] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10User-Urbanecm, and 2 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10zeljkofilipin) [10:52:14] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery-Search, 10MediaWiki-extensions-WikibaseRepository, and 3 others: Wikibase and WikibaseLexeme Selenium tests fail when UseCirrus is true in WikibaseCirrusSearch - https://phabricator.wikimedia.org/T224125 (10zeljkofilipin) [10:54:38] (03CR) 10Awight: Run node browser tests in each repo (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [10:59:51] 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10Volans) @hashar another question for you. If I have 2 CRs, chained one on top of another and I +2 both of them because I want to deploy them together, and the first one fails bu... [11:37:35] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) >>! In T223971#5207016, @Krenair wrote: > I wonder if there should be two separate sets of flavours, on... [11:37:49] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) >>! In T223971#5206979, @bd808 wrote: > * cloudvirt1006.eqiad.wmnet has 2x 12 cores/24 threads, 384G of... [11:41:08] 10Phabricator (Upstream), 10Upstream: Email notification received for Phurl object I cannot view due to application policy - https://phabricator.wikimedia.org/T128696 (10Aklapper) 05Open→03Resolved Resolving per last comment. Thanks for the followup! [11:42:40] 10Release-Engineering-Team (Kanban), 10MediaWiki-SWAT-deployments, 10User-zeljkofilipin: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10Ladsgroup) >>! In T223087#5207682, @zeljkofilipin wrote: > I've talked with @Ladsgroup at #wikimedia-hackathon-2019. He's fine with being added to... [11:48:37] 10Phabricator (Upstream), 10Upstream: Task description diffs sometimes don't show all changes - https://phabricator.wikimedia.org/T154667 (10Aklapper) 05Open→03Resolved Cannot reproduce nowadays, https://phabricator.wikimedia.org/T154071#2918712 and https://phabricator.wikimedia.org/T154071#2920748 look as... [12:00:17] (03CR) 10Zfilipin: Add .gitreview (031 comment) [releng/local-charts] - 10https://gerrit.wikimedia.org/r/511114 (owner: 10Tarrow) [12:17:15] 10Project-Admins: Create project for Wiki Techstorm 2019 - https://phabricator.wikimedia.org/T222929 (10Aklapper) 05Resolved→03Open Reopening as there has been some discussion in T206241. @Ecritures: It's best practice not to remove a tag from resolved tasks, as that makes it hard to get a list of tasks tha... [12:32:36] (03CR) 10Awight: "We might have to wait for the node 10 upgrade: T211784" [integration/quibble] - 10https://gerrit.wikimedia.org/r/510709 (https://phabricator.wikimedia.org/T199116) (owner: 10Awight) [12:34:37] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10SBisson) Note that you may see a spike of {T223741} when deploying wmf.6 to group 2. The fix is [[ https://gerrit.wikimedia.org/r/c/mediaw... [12:38:55] (03PS2) 10Hashar: Use composer for Mediawiki REL branches [integration/config] - 10https://gerrit.wikimedia.org/r/511866 (https://phabricator.wikimedia.org/T189560) [12:41:05] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10aborrero) Here is the result: ` root@cloudvirt1006:~# sysbench --test=cpu run sysbench 0.4.12: multi-threaded... [12:58:31] (03CR) 10Hashar: "I am not sure what was wrong in T219749" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/511308 (https://phabricator.wikimedia.org/T222200) (owner: 10Noa wmde) [13:02:41] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Wikimedia-Hackathon-2019, and 2 others: Run browser tests as part of "npm test" of wikidata/query/gui - https://phabricator.wikimedia.org/T222200 (10hashar) Here is an explanation of the issue @Smalyshev encountered on T219749 as I... [13:06:15] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite, 10Wikimedia-Hackathon-2019, and 2 others: Run browser tests as part of "npm test" of wikidata/query/gui - https://phabricator.wikimedia.org/T222200 (10awight) This might also be useful: {T199116} I have a patch for review which will... [13:10:49] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10Agusbou2015) >>! In T220731#5207828, @SBisson wrote: > Note that you may see a spike of {T223741} when deploying wmf.6 to group 2. The fix... [13:12:53] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10SBisson) >>! In T220731#5207924, @Agusbou2015 wrote: >>>! In T220731#5207828, @SBisson wrote: >> Note that you may see a spike of {T223741... [13:13:53] 10Phabricator (Upstream), 10Upstream: Ability to add warnings for users selecting me as Conpherence target - https://phabricator.wikimedia.org/T98332 (10Aklapper) 05Open→03Declined As far as I know there are no plans in upstream to implement this and we don't plan to have a downstream patch for this either... [13:16:40] 10Phabricator (Upstream), 10Upstream: Allow to disable Conpherence notifications - https://phabricator.wikimedia.org/T98333 (10Aklapper) 05Open→03Declined We're discussing enabling Conpherence in T127640 and there are no plans to implement T98333 in Wikimedia Phabricator, hence reflecting reality by declin... [13:17:06] 10Phabricator (Upstream), 10Upstream: Consolidate Conpherence settings into notification settings - https://phabricator.wikimedia.org/T98334 (10Aklapper) 05Open→03Declined We're discussing enabling Conpherence in T127640 and there are no plans to implement T98334 in Wikimedia Phabricator, hence reflecting... [13:18:03] 10Phabricator (Upstream), 10Upstream: Allow admins to delete specific messages from Conpherence - https://phabricator.wikimedia.org/T190276 (10Aklapper) [13:19:05] 10Phabricator, 10Project-Admins: Use Conpherence for project-admins requests - https://phabricator.wikimedia.org/T128528 (10Aklapper) Note that we're discussing disabling Conpherence in T127640. [13:20:46] 10Phabricator (Upstream), 10Upstream: Simple way to start a Conpherence with another Phabricator user - https://phabricator.wikimedia.org/T152652 (10Aklapper) 05Open→03Declined We're discussing disabling Conpherence in T127640 and there are no plans to implement T152652 in Wikimedia Phabricator, hence refl... [13:22:41] 10Phabricator (Upstream), 10Upstream: Conpherence: "Edit Room" does not edit the correct room (after switching from a room I cannot edit to a room I can edit) - https://phabricator.wikimedia.org/T141883 (10Aklapper) p:05Triage→03Lowest [13:22:55] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) @Sbisson sorry I have missed T223741 hotfix ( https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Echo/+/512070/ ). I think I... [13:29:56] 10Release-Engineering-Team (Kanban), 10MediaWiki-SWAT-deployments, 10User-zeljkofilipin: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10zeljkofilipin) @greg @Jdforrester-WMF I vaguely remember one of you edited the template of deployments page. Can you add @Ladsgroup and @Lucas_Werk... [13:31:05] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10Aklapper) >>! In T127640#5176939, @chasemp wrote: > Security team has an ongoing chat with the stewards via conpherence. So far we (Security) have proposed a m... [13:34:03] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) [13:25:28] !log hashar@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.6 [13:59:58] 10Phabricator (Upstream), 10Upstream: Public log for Phabricator user account role change (e.g. deactivation, admin rights) - https://phabricator.wikimedia.org/T102576 (10epriestley) Yeah, spam/abuse are pretty clear-cut (and probably 90%+ of our upstream disables are for those reasons too), but we've also had... [14:07:03] (03CR) 10Alaa Sarhan: "> Patch Set 2:" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/511308 (https://phabricator.wikimedia.org/T222200) (owner: 10Noa wmde) [14:29:36] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) [14:29:58] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T220731 (10hashar) 05Open→03Resolved Looks like 1.34.0-wmf.6 is mostly fine. [14:30:21] (03CR) 10Alaa Sarhan: Change Dockerfile and wikidata.yml to support running browser tests for the query service gui (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/511308 (https://phabricator.wikimedia.org/T222200) (owner: 10Noa wmde) [14:48:03] (03CR) 10Jforrester: [C: 03+1] Add .gitreview [releng/local-charts] - 10https://gerrit.wikimedia.org/r/511114 (owner: 10Tarrow) [14:48:15] (03CR) 10Jforrester: [C: 03+1] "I think it'll work either way." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/511114 (owner: 10Tarrow) [14:51:33] 10Project-Admins: Create project for Wiki Techstorm 2019 - https://phabricator.wikimedia.org/T222929 (10Ecritures) I myself would be happiest to have for example a 'Wiki Techstorm - 2019' board and keep the old board (and rename that to 'Wiki Techstorm 2018'). In that case it would be nice if with a mass edit th... [14:53:25] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) [14:53:35] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) Thanks! So the benchmarks on the real hardware looks similar to what I get on the instance. I guess tha... [14:55:53] (03CR) 10Brennen Bearnes: [C: 03+1] Add .gitreview [releng/local-charts] - 10https://gerrit.wikimedia.org/r/511114 (owner: 10Tarrow) [14:56:13] 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10BBlack) A few thoughts: * None of our CI on ops/puppet provides very strong guarantees of correctness regardless of what we do here. It's possible for the automatic CI to succ... [15:01:05] 10Continuous-Integration-Config, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10BBlack) One more: * Automatic merges are not fool-proof in general. This is clear in the separate-file case (e.g. changing two separate manifests in two seemingly-unrelated mo... [15:04:46] (03PS3) 10Alaa Sarhan: Change Dockerfile and wikidata.yml to support running browser tests for the query service gui [integration/config] - 10https://gerrit.wikimedia.org/r/511308 (https://phabricator.wikimedia.org/T222200) (owner: 10Noa wmde) [15:05:55] 10Release-Engineering-Team (Kanban), 10MediaWiki-SWAT-deployments, 10User-zeljkofilipin: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10Jdforrester-WMF) Done: https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=1827204&oldid=1827175 [15:07:03] (03CR) 10Alaa Sarhan: "So yeah this is becoming ugly .. we definitely need to find a way to configure what arguments are being passed to the `dokcer run` command" [integration/config] - 10https://gerrit.wikimedia.org/r/511308 (https://phabricator.wikimedia.org/T222200) (owner: 10Noa wmde) [15:07:22] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) [15:10:19] 10Release-Engineering-Team (Kanban), 10MediaWiki-SWAT-deployments, 10User-zeljkofilipin: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10zeljkofilipin) Thanks! 🎉 [15:13:06] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) Ran it on a couple production host and two of my machines at home: | Host | Model |... [15:19:33] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-codesearch, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10thcipriani) 05Open→03Resolved a:03thcipriani >>! In T221026#5206626, @hashar wrote: > Regarding the `SendEmail` thread, it took a while to... [15:19:38] 10Release-Engineering-Team (Kanban), 10MediaWiki-SWAT-deployments, 10User-zeljkofilipin: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10greg) 05Open→03Resolved a:05zeljkofilipin→03Jdforrester-WMF Thanks @Jdforrester-WMF for responding here :) [15:26:22] 10MediaWiki-Codesniffer, 10Code-Health: Create a policy on importing global classes - https://phabricator.wikimedia.org/T223646 (10dbarratt) [15:59:04] 10Phabricator (Upstream), 10Upstream: Using "Sign with MFA", for unchanged actions, the "The actions you are taking have no effect" dialog is shown twice - https://phabricator.wikimedia.org/T224133 (10mmodell) seems like a legit upstream bug [16:10:18] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10aezell) As an Engineering Manager I need to be able to admin projects for Community Tech to help manage our sprint boards. [16:11:06] 10Phabricator (Upstream), 10Upstream: Using "Sign with MFA", for unchanged actions, the "The actions you are taking have no effect" dialog is shown twice - https://phabricator.wikimedia.org/T224133 (10epriestley) Yep, this reproduces for me. (Although I think the behavior is an infinite loop of prompts, not ju... [16:14:48] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) [16:15:30] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) Phabricator has been switched to phab1003 as the prod server now and that meant: - php 5 to... [16:16:31] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) I am thinking now we could make the process easier and just keep phab1003 as the prod server... [16:17:19] 10Phabricator, 10Performance: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T125357 (10Dzahn) >>! In T125357#4804491, @Paladox wrote: > We increased the max_execution_time to 30s in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477595/ f... [16:17:59] 10Phabricator, 10Performance: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded - https://phabricator.wikimedia.org/T125357 (10Paladox) 05Open→03Resolved a:03Paladox Yup! It loads now! [16:18:28] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10mmodell) @dzahn: agreed. I don't know who should decide if we keep phab1001 or return it. We've pro... [16:19:50] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) I will bring this up in my next subteam discussion meeting which should be in a week. Until... [16:55:03] every mobileapps CI test requiring an external HTTP request is now 504ing [16:55:04] 10Phabricator (Upstream), 10Upstream: Using "Sign with MFA", for unchanged actions, the "The actions you are taking have no effect" dialog is shown twice - https://phabricator.wikimedia.org/T224133 (10Hamishcn) >>! In T224133#5208469, @epriestley wrote: > Yep, this reproduces for me. (Although I think the beha... [16:55:08] seems suspicious [16:56:03] it's happening in both npm-node-6-docker and mobileapps-periodic-test [17:11:29] 10Release-Engineering-Team (Kanban), 10Wikimedia-Hackathon-2019, 10Discovery-Search (Current work), 10Patch-For-Review, 10User-zeljkofilipin: The first Selenium test for WikibaseCirrusSearch - https://phabricator.wikimedia.org/T223774 (10EBernhardson) [17:22:29] (03PS1) 10Dduvall: pipeline: Fix node labels for publish configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 [17:22:53] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 (owner: 10Dduvall) [17:27:38] (03PS2) 10Dduvall: pipeline: Fix node labels for publish configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 [17:28:36] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 (owner: 10Dduvall) [17:29:39] 10Release-Engineering-Team (Kanban), 10Wikimedia-Blog-Content: [Technical Debt Series]How to remove Technical Debt - https://phabricator.wikimedia.org/T175183 (10Jrbranaa) 05Stalled→03Declined [17:29:59] 10Release-Engineering-Team (Kanban), 10Wikimedia-Blog-Content: [Technical Debt Series]Avoiding New Technical Debt - https://phabricator.wikimedia.org/T175184 (10Jrbranaa) 05Stalled→03Declined [17:30:08] (03PS3) 10Dduvall: pipeline: Fix node labels for publish configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 [17:30:52] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 (owner: 10Dduvall) [17:32:02] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix node labels for publish configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 (owner: 10Dduvall) [17:32:24] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 (owner: 10Dduvall) [17:32:26] (03Merged) 10jenkins-bot: pipeline: Fix node labels for publish configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 (owner: 10Dduvall) [17:32:50] (03CR) 10jenkins-bot: pipeline: Fix node labels for publish configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512197 (owner: 10Dduvall) [17:36:26] 10Beta-Cluster-Infrastructure: Request to be a interface-admin on zh Beta Cluster - https://phabricator.wikimedia.org/T224204 (10Krenair) 05Open→03Resolved a:03Krenair [17:58:02] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Reading-Infrastructure-Team-Backlog, 10Services: All mobileapps CI tests requiring external HTTP requests are throwing 504s - https://phabricator.wikimedia.org/T224239 (10Mholloway) [17:58:18] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Reading-Infrastructure-Team-Backlog, 10Services: All mobileapps CI tests requiring external HTTP requests are throwing 504s - https://phabricator.wikimedia.org/T224239 (10Mholloway) p:05Triage→03High [18:10:59] 10Phabricator: Create custom data entry form for ;WMF-CTO-Admin-Sensitive - https://phabricator.wikimedia.org/T179617 (10Aklapper) > = Visibility > The form should be be visible to all members of #WMF-CTO-Admin-Sensitive via the global {icon plus}Create Task menu. I don't see how that is possible by default. A... [18:13:36] 10Phabricator (Upstream), 10Upstream: Merging a task notifies other subscribers twice in web UI - https://phabricator.wikimedia.org/T154322 (10Aklapper) [18:18:10] (03PS1) 10Dduvall: pipeline: Fix bad references to publish config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512207 [18:18:54] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512207 (owner: 10Dduvall) [18:19:14] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix bad references to publish config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512207 (owner: 10Dduvall) [18:19:39] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512207 (owner: 10Dduvall) [18:19:41] (03Merged) 10jenkins-bot: pipeline: Fix bad references to publish config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512207 (owner: 10Dduvall) [18:20:06] (03CR) 10jenkins-bot: pipeline: Fix bad references to publish config [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512207 (owner: 10Dduvall) [18:20:08] (03PS1) 10Umherirrender: [GlobalUserPage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/512208 [18:35:19] ahoy. is there any change that the SWAT deployment right now will happen? [18:35:22] chance* [18:53:05] (03PS1) 10Dduvall: pipeline: Fix publish config property references [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512211 [18:53:54] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512211 (owner: 10Dduvall) [18:54:06] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix publish config property references [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512211 (owner: 10Dduvall) [18:54:28] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512211 (owner: 10Dduvall) [18:54:30] (03Merged) 10jenkins-bot: pipeline: Fix publish config property references [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512211 (owner: 10Dduvall) [18:54:51] (03CR) 10jenkins-bot: pipeline: Fix publish config property references [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512211 (owner: 10Dduvall) [19:01:26] (03PS1) 10Dduvall: pipeline: Fix reference to publish.image.id [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512218 [19:01:34] (03PS1) 10Thcipriani: clear_message_blobs: call once per wikiversions [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) [19:01:53] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512218 (owner: 10Dduvall) [19:02:06] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix reference to publish.image.id [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512218 (owner: 10Dduvall) [19:03:23] (03PS2) 10Thcipriani: clear_message_blobs: call once per wikiversion [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) [19:04:19] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [19:04:38] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512218 (owner: 10Dduvall) [19:04:40] (03Merged) 10jenkins-bot: pipeline: Fix reference to publish.image.id [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512218 (owner: 10Dduvall) [19:04:51] 10Phabricator (Upstream), 10Upstream: Using "Sign with MFA", for unchanged actions, the "The actions you are taking have no effect" dialog is shown twice - https://phabricator.wikimedia.org/T224133 (10epriestley) This should be resolved by . [19:05:30] (03CR) 10jenkins-bot: pipeline: Fix reference to publish.image.id [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512218 (owner: 10Dduvall) [19:06:30] (03PS3) 10Thcipriani: clear_message_blobs: call once per wikiversion [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) [19:07:19] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [19:09:34] (03PS1) 10Dduvall: pipeline: Fix references to config.exports in exports step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512221 [19:10:09] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/512219 (https://phabricator.wikimedia.org/T222539) (owner: 10Thcipriani) [19:10:29] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512221 (owner: 10Dduvall) [19:10:37] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix references to config.exports in exports step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512221 (owner: 10Dduvall) [19:11:02] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512221 (owner: 10Dduvall) [19:11:04] (03Merged) 10jenkins-bot: pipeline: Fix references to config.exports in exports step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512221 (owner: 10Dduvall) [19:11:37] (03CR) 10jenkins-bot: pipeline: Fix references to config.exports in exports step [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512221 (owner: 10Dduvall) [19:30:00] (03PS1) 10Dduvall: pipeline: Fix config.exports iteration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512226 [19:30:26] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512226 (owner: 10Dduvall) [19:30:54] (03CR) 10Dduvall: [C: 03+2] pipeline: Fix config.exports iteration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512226 (owner: 10Dduvall) [19:31:17] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512226 (owner: 10Dduvall) [19:31:19] (03Merged) 10jenkins-bot: pipeline: Fix config.exports iteration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512226 (owner: 10Dduvall) [19:31:34] ^ thcipriani not being able to test all PipelineStage methods locally is annoying :( [19:31:43] (03CR) 10jenkins-bot: pipeline: Fix config.exports iteration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/512226 (owner: 10Dduvall) [19:33:09] but i think with the right degree of mocking it might be possible [19:37:45] (03PS5) 10Dduvall: pipeline: Include .pipeline/config.yaml [blubber] - 10https://gerrit.wikimedia.org/r/511784 [19:38:20] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [blubber] - 10https://gerrit.wikimedia.org/r/511784 (owner: 10Dduvall) [20:20:40] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Reading-Infrastructure-Team-Backlog, 10Services: All mobileapps CI tests requiring external HTTP requests are throwing 504s - https://phabricator.wikimedia.org/T224239 (10thcipriani) hrm. I can recreate the test failures in a conta... [20:24:23] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Reading-Infrastructure-Team-Backlog, 10Services: All mobileapps CI tests requiring external HTTP requests are throwing 504s - https://phabricator.wikimedia.org/T224239 (10Mholloway) I'm now seeing this locally after installing fres... [20:26:33] (03CR) 10Sarhan: "This thing https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/jjb/oojs.yaml#7 looks promosing. Is this `script` th" [integration/config] - 10https://gerrit.wikimedia.org/r/511308 (https://phabricator.wikimedia.org/T222200) (owner: 10Noa wmde) [20:34:51] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Old cloudvirt (with Intel Xeon) are twice slower than new ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 (10hashar) Eventually I found a MediaWiki application server with a CPU older than the one on cloudvirt1006: `mw21... [20:37:09] I'm trying to put together developer build instructions for Kask, that use blubber to create a container. Does that seem like the right approach? If so, which version of blubber should I be using? [20:37:48] It seems OK with our blubber.yaml in production, but when I build from master it gets angry about parsing it [20:44:21] mdholloway: around? [20:45:00] mdholloway: your mobile app issue, I think that is due to swagger-router upgraded from 0.7.2 to 0.7.3 [20:45:04] at least based on the diff [20:45:29] aha... [20:46:16] mdholloway: I have "just" diffed the console output of the periodic job ;] [20:46:40] so I blame upstream [20:46:46] hashar: well spotted [20:46:57] i'll test, but i'm pretty sure you're right [20:47:25] yeah +2 on trying to reproducing. It is late for me [20:47:32] but from the diff, that is the only thing that has changed [20:47:45] yes, that is the culprit [20:48:01] maaaagic [20:48:35] and thank you to have filled a task! [21:10:54] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-codesearch, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10hashar) @thcipriani now deserves some well deserved time off after all the madness that task has caused :-] //As a side note, we now have JavaM... [21:15:31] urandom: missed your message earlier, blubber 0.7.0 should work for you, that's the last version we released into production. Should be available: https://releases.wikimedia.org/blubber/ -- master is a bit ahead and we need to bump the version [21:16:25] er...sorry..0.6.0 [21:16:27] thcipriani: is there no tag for 0.7.0? [21:16:31] oh... [21:16:40] https://wikitech.wikimedia.org/wiki/Blubber/Download [21:16:44] yeah, that one had a build error that I made no effort to suss out [21:18:01] OK, I'll just stick with prebuilt binaries [21:18:06] thcipriani: thanks! [21:18:09] sure thing [21:34:05] mutante and twentyafterfour https://grafana.wikimedia.org/d/000000587/phabricator?orgId=1&from=now-24h&to=now now shows phab1003! [21:34:37] cool paladox :) thx [21:35:06] 10Continuous-Integration-Config, 10Parsoid-PHP: Parsoid/PHP post-merge doc publishing task needs updating to use PHP 7.2 - https://phabricator.wikimedia.org/T219319 (10Jdforrester-WMF) [21:36:18] wow mutante well https://grafana.wikimedia.org/d/000000587/phabricator?orgId=1&from=now-2d&to=now&panelId=7&fullscreen looks much healther then the image linked here https://phabricator.wikimedia.org/T182832#5208456 for phab1001 [21:39:08] paladox: indeed :) i am glad we can still confirm that issue is gone [21:40:34] paladox: feel free to link that new image too . it's nice [21:40:41] ok [21:40:48] (03CR) 10Hashar: [C: 04-1] "You have a good point, I have not taken in account mediawiki/vendor, then if I look at jobs triggered for a REL1_33 https://gerrit.wikimed" [integration/config] - 10https://gerrit.wikimedia.org/r/511866 (https://phabricator.wikimedia.org/T189560) (owner: 10Hashar) [21:41:57] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Paladox) Updated image for phab1003: {F29226685} It looks m... [21:42:11] done [21:51:09] thx [21:56:27] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-codesearch, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10Paladox) I agree with @hashar, @thcipriani did a fantastic job at tuning gerrit! [22:08:46] 10Gerrit, 10Release-Engineering-Team: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10thcipriani) ## Disk There are roughly 6,900 gerrit user accounts. From the gerrit config page (https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#cache_names) > Sess... [22:28:45] 10Release-Engineering-Team (Backlog), 10Scap: When scap deploy is aborted it should say so in the log - https://phabricator.wikimedia.org/T199388 (10greg) [22:32:42] 10Deployments, 10Release-Engineering-Team (Kanban), 10User-greg: Outline options for managing incident reports creation, follow-ups, and analysis - https://phabricator.wikimedia.org/T208632 (10greg) [22:32:44] 10Release-Engineering-Team (Kanban): Develop set of metrics to assess incident reports/post mortems - https://phabricator.wikimedia.org/T206622 (10greg) 05Open→03Resolved a:03zeljkofilipin This was done as a part of https://phabricator.wikimedia.org/phame/post/view/128/incident_documentation_an_unexpected_... [22:38:55] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [22:45:02] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell: Publish the source for phabricator-antivandalism - https://phabricator.wikimedia.org/T202080 (10greg) Any opinions on this @chasemp or others from Security Team? [22:46:20] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10User-zeljkofilipin, 10good first bug: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10greg) @zeljkofilipin all the patches listed in the description are merged, are we good here? [22:52:17] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T220732 (10greg) a:03zeljkofilipin [22:52:31] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.34.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T220733 (10greg) a:03zeljkofilipin [23:08:28] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10User-Urbanecm, and 2 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10greg) Yup, +1. Thanks @Urbanecm. Now for some training with @zeljkofilipin :) [23:09:14] 10Release-Engineering-Team (Kanban), 10Operations, 10SRE-Access-Requests, 10User-Urbanecm, and 2 others: Requesting access to production for SWAT deploy for Urbanecm - https://phabricator.wikimedia.org/T192830 (10greg) a:05greg→03None