[00:11:01] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281625 (10mmodell) This seems related https://bugs.chromium.org/p/gerrit/issues/detail?id=5252 [00:19:20] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281626 (10mmodell) this change prevents duplicate emails: https://gerrit-review.googlesource.com/c/gerrit/+/170091 [00:20:43] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281628 (10Paladox) They reverted it for performance reasons [00:20:59] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281630 (10kaldari) I wonder if this might be related to the old T152640 bug (which also affected me). [00:22:30] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281632 (10Paladox) Yes that seems related as 2.15 uses the index again I think [00:34:17] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281651 (10mmodell) I think we may need to use [[ https://gerrit-documentation.storage.googleapis.com/Documentation/2.13.5/config-gerrit.html#ldap.localUsernameToLowerCase | ld... [00:40:04] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281652 (10Paladox) We already set that :) Just ssh is case sensitive and logins are case insensitive. [01:55:11] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281679 (10demon) How about https://gerrit-review.googlesource.com/c/gerrit/+/170693? [02:36:44] Project beta-scap-eqiad build #211723: 04FAILURE in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/211723/ [02:47:49] Yippee, build fixed! [02:47:50] Project beta-scap-eqiad build #211724: 09FIXED in 4 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/211724/ [02:49:12] 10Release-Engineering-Team, 10Multi-Content-Revisions (MCR-SDC phase 1), 10User-Addshore: Deploy some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4281719 (10thcipriani) >>! In T196585#4264878, @Addshore wrote: >>>! In T196585#4264833, @greg wrote: >... [05:57:46] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4281783 (10Marostegui) 05Open>03Resolved a:03mmodell This is now back to normal values: https://grafana.wikimedia.org/dashboard/db/mysql?panelId=2&fullscreen&orgId=1&var... [06:11:14] 10Scap, 10Cassandra, 10Maps-Sprint, 10Operations: cassandra/metrics-collector does not deploy with scap on a new install - https://phabricator.wikimedia.org/T197159#4281799 (10Gehel) Looking on `deploy1001`, I see that `/srv/deployment/cassandra/metrics-collector/.git/DEPLOY_HEAD` also has a reference to `... [06:11:32] 10Release-Engineering-Team, 10Scap, 10Cassandra, 10Maps-Sprint, 10Operations: cassandra/metrics-collector does not deploy with scap on a new install - https://phabricator.wikimedia.org/T197159#4281800 (10Gehel) [07:08:13] 10Continuous-Integration-Infrastructure, 10Operations-Software-Development: integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth] - https://phabricator.wikimedia.org/T197207#4281868 (10hashar) [07:09:21] 10Continuous-Integration-Infrastructure, 10Operations-Software-Development: integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth] - https://phabricator.wikimedia.org/T197207#4281879 (10hashar) ``` $ keyholder status keyholder-agent: active - The agent has no identities. key... [07:09:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations-Software-Development: integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth] - https://phabricator.wikimedia.org/T197207#4281880 (10hashar) a:03hashar [07:12:11] PROBLEM - Puppet errors on integration-cumin is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:13:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations-Software-Development: integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth] - https://phabricator.wikimedia.org/T197207#4281888 (10hashar) 05Open>03Resolved Found the passphrase... [07:15:54] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281902 (10Aklapper) [07:32:13] RECOVERY - Puppet errors on integration-cumin is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:23] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281936 (10hashar) ``` $ ssh -p 29418 gerrit.wikimedia.org 'gerrit gsql' gerrit> select * from account_external_ids where email_address='jbranaa@wikimedia.... [07:36:11] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281937 (10Paladox) @hashar that is old dat :), it’s stored in All-Users now which you have to clone a repo. [07:38:58] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281944 (10hashar) Yup I got that Gerrit is now relying on NoteDB for the accounts data, but at least the database give us a snapshot of the state pre migr... [07:39:51] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281945 (10Paladox) Ah I see, if it’s not too much trouble yeh we could fix it :). [07:40:16] !log Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01| T197207 [07:40:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:40:19] T197207: integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth] - https://phabricator.wikimedia.org/T197207 [07:40:37] !log Cleaned up bunch of Docker containers and images from the CI slave-docker-* instances [07:40:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:41:05] (03PS1) 10PleaseStand: ForbiddenFunctionsSniff: fix typo of "passthru" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/440276 [07:46:00] yeah mediawiki/core received a spec for recent changes: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/439878 \o/ [07:47:12] This seems like the index again [07:51:54] hashar: would you be able to add any debugging info you get to https://bugs.chromium.org/p/gerrit/issues/detail?id=9256&q=&colspec=ID%20Type%20Stars%20Milestone%20Status%20Priority%20Owner%20Summary please? :) [07:51:59] For ekempin [08:07:58] hashar: yup :D [08:08:07] hashar: MORE BROWSER TESTS nom nom nom [08:08:23] paladox: well I would say hold a bit and don't spam too much on the issues/tasks :] [08:08:52] paladox: seems to me Gerrit is mostly working, with the annoyance that some accounts cant be edited, it is probably not too much of an issue [08:09:12] addshore: yeah and an issue in AntiSpoof has been fixed as well which unlock more repositories for a migration to Quibble \o/ [08:09:20] addshore: anomie is a pro [08:09:59] woo! [08:10:05] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4282025 (10Paladox) Upstream has wrote a reply https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c10 [08:10:33] paladox: someone from google came with some questions at https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c10 I think we should copy paste that to our phabricator task and do the investigation on phabricator. Then when we have a thorough analysis we can come back to Gerrit upstream [08:10:35] hashar: ok [08:10:42] Ah ok [08:10:44] paladox: else we risk to exhaust them with lot of investigations details etc :] [08:10:50] Yeh [08:11:09] paladox: their time is precious, so surely we want to come up with a fairly limited scope / great detailed issue [08:11:14] hashar: https://grafana.wikimedia.org/dashboard/db/ci-docker-jobs?panelId=2&fullscreen&orgId=1&from=now-2y&to=now&var-Summarize=7d [08:11:18] :D [08:12:47] Yep [08:16:29] paladox: I replied on the issue saying we will come back later :] [08:17:12] but at firsdt glance, I think notedb is fine [08:17:30] it is just that the reviewdb was wrong having dupe emails/accounts [08:17:31] we will see [08:17:54] addshore: yup I am using your graph as a metric for the progression [08:18:35] we would need the mwext-phpunit-coverage-patch job to be converted [08:18:54] and I will attempt to get the mediawiki-extensions-* converted this week (at least as an experimental job) [08:20:40] Ah ok :) [08:21:09] hashar: we would need to reindex an account after changing it in that ref too [08:21:16] And thanks for working on it :) [08:23:23] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: mediawiki-core-qunit-selenium-jessie sometimes fails for extensions - https://phabricator.wikimedia.org/T195496#4282052 (10hashar) [08:23:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikimedia-log-errors (Jenkins Failure): selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4282053 (10hashar) [08:23:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Math, and 4 others: MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4282048 (10hashar) 05Open>03Resolve... [08:25:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikimedia-log-errors (Jenkins Failure): selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4282055 (10hashar) 05Open>03Resolved a:03Anomie As... [08:25:57] zeljkof: marking the selenium tasks flappiness fixed ( https://phabricator.wikimedia.org/T189762 https://phabricator.wikimedia.org/T191537 ) [08:26:10] hashar: yeah! [08:26:42] hashar: for later but is kaldari referenced twice in the repo? [08:26:42] I'll move the math job then to test and gate-and-submit pipelines, it was moved to experimental because it was failing [08:28:12] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: mediawiki-core-qunit-selenium-jessie sometimes fails for extensions - https://phabricator.wikimedia.org/T195496#4282077 (10hashar) [08:28:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Math, 10User-zeljkofilipin: mediawiki-core-qunit-selenium-jessie tests for math broken - https://phabricator.wikimedia.org/T195206#4282072 (10hashar) 05Open>03Resolved That is most probably related to T189762 T191537 . The... [08:28:59] PROBLEM - Free space - all mounts on deployment-mediawiki06 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki06.diskspace.root.byte_percentfree (<11.11%) [08:29:40] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#4282084 (10hashar) [08:29:45] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: mediawiki-core-qunit-selenium-jessie sometimes fails for extensions - https://phabricator.wikimedia.org/T195496#4228968 (10hashar) 05Open>03Resolved Fixed via T189762 and T191537 The test ends up being logged out at some poin... [08:34:08] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4282105 (10hashar) Lets do the analysis on our Phabricator task based on upstream questions. Then we can come back to them with a detailed bug report. At... [08:36:10] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [08:39:01] RECOVERY - Free space - all mounts on deployment-mediawiki06 is OK: OK: All targets OK [08:45:38] !log maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=eswiki --fix (64 broken pagelinks resolved, no remaining conflicts) [08:45:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:49:56] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:59:54] hi mutante do you think https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/439482/ is +2able? [09:09:46] (03CR) 10Addshore: [C: 032] ForbiddenFunctionsSniff: fix typo of "passthru" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/440276 (owner: 10PleaseStand) [09:10:52] (03Merged) 10jenkins-bot: ForbiddenFunctionsSniff: fix typo of "passthru" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/440276 (owner: 10PleaseStand) [09:11:19] (03CR) 10jenkins-bot: ForbiddenFunctionsSniff: fix typo of "passthru" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/440276 (owner: 10PleaseStand) [09:24:54] RECOVERY - Puppet errors on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:28:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4282261 (10hashar) [09:34:57] (03PS1) 10Hashar: Migrate to Quibble extensions blocked by AntiSpoof [integration/config] - 10https://gerrit.wikimedia.org/r/440298 (https://phabricator.wikimedia.org/T195020) [09:36:02] (03CR) 10Hashar: [C: 032] Migrate to Quibble extensions blocked by AntiSpoof [integration/config] - 10https://gerrit.wikimedia.org/r/440298 (https://phabricator.wikimedia.org/T195020) (owner: 10Hashar) [09:37:29] 10Continuous-Integration-Infrastructure, 10AntiSpoof, 10Patch-For-Review: AntiSpoof extension does not pass quibble-vendor-mysql-php70-docker - https://phabricator.wikimedia.org/T195020#4282321 (10hashar) @Umherirrender seems like using binary has a database character set fix the test. Thus I am not sure wh... [09:38:51] (03Merged) 10jenkins-bot: Migrate to Quibble extensions blocked by AntiSpoof [integration/config] - 10https://gerrit.wikimedia.org/r/440298 (https://phabricator.wikimedia.org/T195020) (owner: 10Hashar) [09:43:11] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4282339 (10hashar) [09:45:11] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-Other: EditSubpages extension fails mediawiki/core webdriver.io tests - https://phabricator.wikimedia.org/T196436#4256182 (10hashar) [09:46:21] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-extensions-Other, 10User-zeljkofilipin: EditSubpages extension fails mediawiki/core webdriver.io tests - https://phabricator.wikimedia.org/T196436#4282350 (10zeljkofilipin) [09:50:47] hashar: ah so I was wrong about it being a duplicate external id [09:50:56] Rather it is connected to multiple accounts [10:29:27] 10MediaWiki-Releasing: Extension:Nuke is absent in MediaWiki 1.31.0 tarball - https://phabricator.wikimedia.org/T197229#4282489 (10edwardspec) [10:31:00] 10MediaWiki-Releasing, 10MW-1.31-release: Extension:Nuke is absent in MediaWiki 1.31.0 tarball - https://phabricator.wikimedia.org/T197229#4282499 (10Legoktm) [10:37:04] 10Release-Engineering-Team, 10Scap, 10Cassandra, 10Maps-Sprint, 10Operations: cassandra/metrics-collector does not deploy with scap on a new install - https://phabricator.wikimedia.org/T197159#4280346 (10Joe) Yes, giving a simple `scap deploy --init` in the directory on deployment1001 was enough to fix t... [10:37:19] 10Release-Engineering-Team, 10Scap, 10Cassandra, 10Maps-Sprint, 10Operations: cassandra/metrics-collector does not deploy with scap on a new install - https://phabricator.wikimedia.org/T197159#4282505 (10Joe) 05Open>03Resolved a:03Joe [10:48:05] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#3831274 (10zeljkofilipin) [10:48:16] 10MediaWiki-Releasing, 10MW-1.31-release, 10User-MarcoAurelio: Extension:Nuke is absent in MediaWiki 1.31.0 tarball - https://phabricator.wikimedia.org/T197229#4282527 (10MarcoAurelio) a:03MarcoAurelio It's missing on .gitmodules for REL1_31. Fixing. [10:53:45] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#4282533 (10zeljkofilipin) @hashar this task could be a good place to discus your concerns from [[ https://gerrit.wik... [11:06:59] (03PS1) 10Zfilipin: Revert "Run qunit-selenium job for Math in experimental pipeline" [integration/config] - 10https://gerrit.wikimedia.org/r/440325 (https://phabricator.wikimedia.org/T162455) [11:07:55] /nick hasharAway [11:10:27] fail :P [11:11:11] 10Release-Engineering-Team (Someday), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Run Selenium tests using Firefox - https://phabricator.wikimedia.org/T161697#3139916 (10zeljkofilipin) [11:30:14] 10MediaWiki-Releasing, 10MW-1.31-release, 10User-MarcoAurelio: Extension:Nuke is absent in MediaWiki 1.31.0 tarball - https://phabricator.wikimedia.org/T197229#4282588 (10MarcoAurelio) 05Open>03Resolved [11:40:40] 10MediaWiki-Releasing, 10MW-1.31-release, 10User-MarcoAurelio: Extension:Nuke is absent in MediaWiki 1.31.0 tarball - https://phabricator.wikimedia.org/T197229#4282489 (10Legoktm) This'll be included in 1.31.1 (unsure exactly when that'll be). [11:47:03] hashar: it seems that we are getting -1 from jenkins-bot on all operations/puppet changes currently [11:47:06] 11:45:35 rake aborted! [11:47:06] 11:45:35 NoMethodError: undefined method `positive?' for 0:Fixnum [11:47:06] 11:45:35 /srv/workspace/puppet/rake_modules/git.rb:47:in `uncommitted_changes?' [11:47:10] 11:45:35 /srv/workspace/puppet/rake_modules/taskgen.rb:128:in `block in setup_wmf_styleguide_delta' [11:47:13] 11:45:35 /srv/workspace/puppet/.bundle/ruby/2.1.0/gems/rake-12.0.0/exe/rake:27:in `' [11:47:16] 11:45:35 Tasks: TOP => test => wmf_styleguide_delta [11:50:49] (03CR) 10Physikerwelt: "maybe link to the original commit I50325f46ef0c3e562e37737dc88b65ed583a9378" [integration/config] - 10https://gerrit.wikimedia.org/r/440325 (https://phabricator.wikimedia.org/T162455) (owner: 10Zfilipin) [11:56:40] hashar: you can ignore it. joe is already fixing it [11:57:01] due to a change he made in wmf-styleguide [13:06:55] i'm going to run a maintenance script on deployment-tin.deployment-prep to fix the GettingStarted extension's automated browsers tests. cool? [13:15:02] done [13:30:42] mutante: are you a gerrit admin? [13:30:57] or anyone else here? :D [13:31:04] hashar? :D [13:32:07] addshore: what's up? [13:32:19] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/405015/ is set to WIP (some new gerrit thing) [13:32:35] and the owner can't click the "start review" button because there isn't one [13:32:40] wondering if a gerrit admin can see said button [13:32:47] 10Gerrit: No way to mark patches as not WIP - https://phabricator.wikimedia.org/T197238#4282784 (10Tgr) [13:33:10] 10Gerrit: No way to mark patches as not WIP - https://phabricator.wikimedia.org/T197238#4282794 (10Tgr) p:05Triage>03Unbreak! [13:33:29] Dzahn [13:33:30] Set Ready For Review [13:33:36] addshore: yes, i saw it and clicked it [13:33:39] YAY! [13:34:00] paladox: ^ [13:35:20] that is a very odd issue [13:51:28] mutante: could you so the same for https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/425287/ please? [13:54:32] 10Gerrit: No way to mark patches as not WIP - https://phabricator.wikimedia.org/T197238#4282838 (10Addshore) I got @Dzahn to fix our patches for now. I guess as a gerrit admin he can see this button on everyone's patches. I'm in the gerrit repo creators group, which i guess means I can for some reason see the b... [13:54:41] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4282841 (10Addshore) [13:56:32] addshore: that's a mediawiki core change, did you really mean that? then i cant [13:56:51] (03CR) 10Zfilipin: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [13:57:01] yes, then you cant? but just already did one :P [13:57:12] "start review" just removed the WIP flag, see https://phabricator.wikimedia.org/T197238#4282794 [13:57:19] oh.. _that_ [13:57:22] :D [13:57:29] i thought merging like the phab change :p [13:57:40] nope! *didn't see a phab change* [13:57:50] done [13:57:51] I think that was in #wikimedia-operations ;) [13:57:53] thanks! [13:57:56] yes, it was [13:58:39] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Epic, 10Tracking, 10User-zeljkofilipin: Selenium framework improvements - https://phabricator.wikimedia.org/T182986#4282857 (10zeljkofilipin) [13:58:45] 10Release-Engineering-Team, 10MediaWiki-Vagrant, 10Browser-Tests, 10User-zeljkofilipin: Running selenium inside Vagrant with xvfb or X11 does not work - https://phabricator.wikimedia.org/T196646#4282856 (10zeljkofilipin) [13:59:09] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10MediaWiki-Vagrant, 10User-zeljkofilipin: Running selenium inside Vagrant with xvfb or X11 does not work - https://phabricator.wikimedia.org/T196646#4264427 (10zeljkofilipin) [14:00:52] addshore: use polygerrit :) [14:00:57] There’s a start review button [14:01:04] It’s big and on the top bar [14:01:07] Hard to miss [14:01:08] paladox: nope, apparently not [14:01:08] :) [14:01:16] i see the button in both views [14:01:18] others do not [14:01:19] addshore: if the change isent yours [14:01:24] Then there’s no button [14:01:25] (03CR) 10jerkins-bot: [V: 04-1] Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [14:01:27] paladox: nope, not for tgr or daniel [14:01:29] That’s been fixed recently [14:01:54] addshore: was the change there’s? [14:01:59] yup [14:02:23] Hmm there should be a button if there the owner of the change [14:02:27] I see the button for all of my wip changes [14:02:28] Or a admin [14:02:29] https://usercontent.irccloud-cdn.com/file/5K64RTT9/image.png [14:02:51] I guess they uploaded a change to someone else’s [14:03:16] That’s been fixed to expand it to project owners [14:08:05] (03CR) 10Zfilipin: "mediawiki-selenium-integration-jessie is failing with `Error: You might be using an older PHP version`. I'll fix it." [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [14:08:37] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4282784 (10Paladox) There is a start review button in polygerrit only the owner of the change can do it. Upstream have changed this so project owners can unmark wips. [14:10:41] addshore: mutante: sorry I was working outside of home without internet connectivity [14:10:46] using good old pen and paper [14:10:47] :D [14:11:50] mutante: as you have admin you can unmark wips! [14:16:12] Anyways that’s not a ubn task :) [14:16:27] Ask a admin to unmark wips if you doint own it [14:16:40] When we upgrade to 2.15.3 this should be fixed [14:16:45] A preference added [14:16:53] And project owners should be able to [14:16:56] Unmark wips [14:17:08] addshore: tgr ^^ [14:17:45] I think it was UBN as we really needed these patches to be able to be merged [14:17:53] perhaps just high now, as its friggin annoying :D [14:18:00] Heh [14:18:17] Well we will need to get 2.15.3 as soon as it’s released it fixes a gc bug [14:18:22] Where it keeps files open [14:20:52] (03PS1) 10Zfilipin: mediawiki-selenium-integration-jessie fails with `Error: You might be using an older PHP version` [integration/config] - 10https://gerrit.wikimedia.org/r/440345 [14:22:33] 10Gerrit, 10Release-Engineering-Team, 10Upstream: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4282915 (10mmodell) @hashar: yeah, essentially the problem is that migration didn't catch the duplicates but now gerrit has changed to enforcing the email... [14:23:21] (03CR) 10Zfilipin: [C: 032] mediawiki-selenium-integration-jessie fails with `Error: You might be using an older PHP version` [integration/config] - 10https://gerrit.wikimedia.org/r/440345 (owner: 10Zfilipin) [14:25:13] (03Merged) 10jenkins-bot: mediawiki-selenium-integration-jessie fails with `Error: You might be using an older PHP version` [integration/config] - 10https://gerrit.wikimedia.org/r/440345 (owner: 10Zfilipin) [14:26:01] !log Reloading Zuul to deploy 04b1c864f3b9d49e9b3882a1fc4c8fbf272c1bd7 [14:26:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:26:27] (03CR) 10Zfilipin: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [14:27:16] (03CR) 10Hashar: [V: 032 C: 032] Archive the CustomPage skin [skins/CustomPage] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440110 (https://phabricator.wikimedia.org/T196429) (owner: 10MarcoAurelio) [14:29:59] hashar: does git submodule update --rebase work for the mediawiki/skins thing? [14:36:09] (03CR) 10Zfilipin: [C: 032] Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [14:38:38] (03Merged) 10jenkins-bot: Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [14:40:19] (03CR) 10jenkins-bot: Caused by switching the form to OOUI. [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [14:42:15] (03PS1) 10Hashar: Migrate AkismetKlik to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440348 (https://phabricator.wikimedia.org/T195831) [14:42:28] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4282947 (10hashar) [14:42:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4282951 (10hashar) [14:43:04] (03CR) 10Hashar: [C: 032] Migrate AkismetKlik to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440348 (https://phabricator.wikimedia.org/T195831) (owner: 10Hashar) [14:43:35] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4255956 (10hashar) [14:44:46] (03Merged) 10jenkins-bot: Migrate AkismetKlik to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440348 (https://phabricator.wikimedia.org/T195831) (owner: 10Hashar) [14:47:06] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4282955 (10Tgr) Most changeset owners do not have that button currently, and many patches have been marked as WIP somehow (compare [[https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MassMessage/+/4... [14:48:15] Tgr did someone hide https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MassMessage/+/4 ? [14:49:00] that link gives me an error [14:49:39] 10Continuous-Integration-Config, 10StructuredDiscussions, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10SQLite: Flow does not depend upon Echo in CI - https://phabricator.wikimedia.org/T185608#4282956 (10kostajh) a:03kostajh [14:49:45] Ah I see now [14:49:49] Copied the wrong link [14:49:51] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MassMessage/+/437064/ [14:50:24] I used the WIP button on that, if that's what you're asking [14:50:29] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4282784 (10MarcoAurelio) Patches you create directly on the Gerrit UI are marked WIP as default, and only the owner of the patch can unmark it as WIP by clicking on the 'Start Review' button. I am not a... [14:50:45] (03PS1) 10Zfilipin: Release patch version 1.8.1 [selenium] - 10https://gerrit.wikimedia.org/r/440349 [14:51:00] tgr and you carnt unmark it as wip? [14:51:18] I can't [14:51:44] curiourly Daniel's patch tgr mentions is not created via the Gerrit UI [14:51:46] Oh I see now tgr [14:51:48] and is marked as wip [14:51:52] Drafts were converted to wips [14:51:59] ah, that explains [14:51:59] As they removed drafts [14:52:03] And split the function [14:52:08] Into private / wips [14:52:12] We disabled private [14:52:34] But it doesn’t explain why Daniel cannot unmark his change at [14:52:34] (03CR) 10Zfilipin: [C: 032] Release patch version 1.8.1 [selenium] - 10https://gerrit.wikimedia.org/r/440349 (owner: 10Zfilipin) [14:52:35] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/421433/ [14:53:36] Hauskatze: I created https://gerrit.wikimedia.org/r/#/c/mediawiki/vagrant/+/440297/ via the Gerrit UI. It was indeed marked automatically as WIP, and I do have a "Start Review" button on that [14:53:40] (03Merged) 10jenkins-bot: Release patch version 1.8.1 [selenium] - 10https://gerrit.wikimedia.org/r/440349 (owner: 10Zfilipin) [14:54:05] tgr: yep, indeed; on patches you own you can mark them wip/not-wip [14:54:16] I insist: on those you own [14:54:30] you cannot up-wip other's patches... at least not no-admins? [14:54:31] Upstream have baked that into your preference now! [14:54:46] Hauskatze: upstream fixed that too so project owners can [14:54:46] s/up-wip/un-wip [14:54:52] When we upgrade to 2.15.2 [14:54:55] 2.15.3 [14:54:57] I mean :) [14:55:03] but only on the old interface, apparently [14:55:17] in polygerrit I still don't see any such option [14:55:20] (03CR) 10jenkins-bot: Release patch version 1.8.1 [selenium] - 10https://gerrit.wikimedia.org/r/440349 (owner: 10Zfilipin) [14:55:24] paladox: good, but with Chad leaving (wtf I learned that today :'( :'( ) who's taking care of the gerrit upgrades? [14:55:44] Releng [14:56:13] so probably a blocker for the polygerrit switch [14:56:31] Tgr blocker? [14:57:05] well, if you can't unWIP you patches in polygerrit, that's a pretty severe UX defect [14:57:09] (03CR) 10Zfilipin: "mediawiki_selenium 1.8.1 is released" [selenium] - 10https://gerrit.wikimedia.org/r/439559 (owner: 10Thiemo Kreuz (WMDE)) [15:02:46] tgr hmm, so everyone can only see it in the old ui [15:02:54] If it was a draft migrated to wip [15:06:43] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4282981 (10Tgr) p:05Unbreak!>03High I was slightly confused: the unWIP option seems to be missing completely from the new UI; on the old UI changeset owners can indeed see it. So this is not that ba... [15:07:08] 10MediaWiki-Codesniffer: IfElseStructureSniff produces Undefined index - https://phabricator.wikimedia.org/T197197#4282985 (10Legoktm) [15:10:17] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4282995 (10Tgr) >>! In T197238#4282981, @Tgr wrote: > on the old UI changeset owners can indeed see it Although @daniel says he doesn't. Maybe there is some difference in what permissions are granted b... [15:19:08] (03PS1) 10Hashar: Migrate two extensions to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440352 (https://phabricator.wikimedia.org/T183512) [15:19:34] (03CR) 10Hashar: [C: 032] Migrate two extensions to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440352 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [15:21:15] (03Merged) 10jenkins-bot: Migrate two extensions to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440352 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [15:23:03] (03PS2) 10Zfilipin: Run Selenium tests for Math [integration/config] - 10https://gerrit.wikimedia.org/r/440325 (https://phabricator.wikimedia.org/T162455) [15:27:58] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4283054 (10mmodell) @kaldari can you try logging into gerrit again? I think I might have it sorted. [15:28:27] hi thcipriani :D [15:28:34] addshore: oh hai [15:28:34] kaldari: can you log in to gerrit now? [15:28:54] addshore: how did the merges to master go? [15:29:01] the last one is going in now [15:29:21] been playing some fun games with gerrit [15:30:12] that doesn't sound fun, really :) [15:30:44] https://memegenerator.net/img/instances/51966536/happy-birthday-gerrit-i-hope-you-get-a-panda.jpg [15:30:57] twentyafterfour ohhh how did you fix it? :) :) [15:31:18] twentyafterfour will need to start the online reindexer for accounts [15:31:21] using the rest api to delete the email from one of the accounts [15:31:33] paladox: I'm not sure about how to do that [15:31:42] twentyafterfour ssh -p 29418 review.example.com gerrit index start accounts [15:31:46] https://gerrit-review.googlesource.com/Documentation/cmd-index-start.html [15:32:02] so for you i think it's ssh -p 29418 20after4@gerrit.wikimedia.org gerrit index start accounts [15:32:13] Nothing to reindex, index is already the latest version [15:32:19] addshore: ok, let me know when everything looks correct on beta cluster and then I can cut a branch (I'll probably just name it wmf.999 to stay on the safe-side of all the tooling) [15:32:19] ok [15:32:33] thcipriani: okay! [15:32:36] 666 ;) [15:32:42] triple digits work? :) [15:33:01] at least one of the regexes was \.\d+ [15:33:12] :) [15:33:43] twentyafterfour that seemed to fix it [15:33:49] no duplicate kaldari in gerrit! [15:35:04] thanks twentyafterfour! [15:35:56] now what other users to fix... [15:36:27] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4283075 (10hashar) [15:36:55] twentyafterfour wondering how did you fix it? [15:37:18] since i am pretty sure i got this wrong with it being a duplicate external id, rather it being a email attached to someone elses account. [15:38:19] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4283080 (10mmodell) @Jrbranaa Can you try as well? Your duplicate account has been disabled and stripped of its email address. [15:38:46] twentyafterfour https://phabricator.wikimedia.org/T138672 [15:38:55] that was a problem before the 2.15 upgrade [15:42:59] paladox: I don't see that user duplicated [15:43:10] ok [15:44:09] i get "Server error: User smccandlish not found" when searching for them [15:44:12] which is strange [15:44:19] i wonder which email he is using [15:49:53] does mah and SMcCandlish share the same email? [15:50:09] nvm [15:51:31] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Document running Selenium tests targeting custom MediaWiki install - https://phabricator.wikimedia.org/T196561#4283120 (10zeljkofilipin) [15:51:34] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994#4283121 (10zeljkofilipin) [15:52:37] https://phabricator.wikimedia.org/T138672#2408607 [15:52:42] twentyafterfour ^^ [15:53:52] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4283126 (10Paladox) >>! In T138672#2408607, @Krenair wrote: > Yes, we know what's causing this i... [15:56:49] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4283129 (10Paladox) >>! In T197238#4282981, @Tgr wrote: > I was slightly confused: the unWIP option seems to be missing completely from the new UI; on the old UI changeset owners can indeed see it. So t... [15:58:13] twentyafterfour is this "gtisza@wikimedia.org" specified twice in there? [15:58:58] paladox: yes [15:59:06] ah ok [15:59:12] twentyafterfour which accounts? :) [15:59:18] that's why i am getting a cannot find that [15:59:19] error [15:59:20] heh [15:59:38] 389 and 2280 [16:00:01] ah [16:00:02] found it [16:00:06] Multimedia [16:00:11] tgr ^^ [16:00:22] you cannot use the same email for the same account! :) [16:07:59] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4283206 (10Paladox) https://gerrit-review.googlesource.com/c/gerrit/+/183851 is the fix which expands it to project owners too so not limited to just admins. [16:08:19] 10Continuous-Integration-Config, 10StructuredDiscussions, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10SQLite: Flow does not depend upon Echo in CI - https://phabricator.wikimedia.org/T185608#4283207 (10kostajh) a:05kostajh>03Legoktm @Legoktm is this still an issue? It looks like this was... [16:13:34] twentyafterfour when you have time could you answer https://bugs.chromium.org/p/gerrit/issues/detail?id=9256#c10 please? :) [16:37:23] addshore: around for addshore/mediawiki-docker-dev question? [16:38:02] I remember it working on my machine, but now that I have tried running it again, followed all instructions, as the last time, no error messages [16:38:32] http://127.0.0.1:8080/ returns 503 Service Temporarily Unavailablenginx/1.13.12 [16:38:38] so something is running [16:39:20] but for http://default.web.mw.localhost:8080/ I get `server not found` [16:41:00] zeljkof: yes [16:41:23] did you get anything interesting in the logs? [16:41:42] while in the mediawiki-docker-dev dir run "docker-compose logs -f" [16:41:46] and make another request [16:42:29] where are logs? [16:43:16] run "docker-compose logs -f" and youll get the stdout of all of the running services / containers [16:43:18] ah, that will create logs, ok [16:43:27] also, what is the output of "docker-compose ps" [16:44:37] addshore: https://phabricator.wikimedia.org/P7259 [16:44:58] okay, that looks fine [16:45:06] anything in the logs? :) [16:45:10] can pastebin them too [16:45:30] addshore: https://phabricator.wikimedia.org/P7260 [16:46:38] I see some requests apparently hitting mediawiki [16:46:38] nginx-proxy_1 | nginx.1 | 127.0.0.1 172.18.0.1 - - [14/Jun/2018:16:35:39 +0000] "GET /wiki/Special:Version HTTP/1.1" 503 214 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0" [16:47:20] but that is only the proxy [16:47:37] I would just try running the "down" command and then the "up" command again, something seems to have gone wrong? [16:47:42] if it doesnt work then we can debug more [16:47:42] that's me trying Special:Version but getting 503 [16:48:04] I think I did down and up already, will try again [16:49:16] down and up did not help, same problem :/ [16:50:27] hmmm [16:50:48] trying on another machine [16:50:56] seems like the nginx proxy cant connect to the mw container for some reason [16:50:58] just to make sure I did not mess up something on this one [16:51:53] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4283359 (10Jrbranaa) >>! In T197083#4283080, @mmodell wrote: > @Jrbranaa Can you try as well? Your duplicate account has been disabled and stripped of its email address. @mmod... [16:54:05] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4283369 (10Paladox) p:05Unbreak!>03High [16:55:07] 10Continuous-Integration-Config, 10StructuredDiscussions, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10SQLite: Flow does not depend upon Echo in CI - https://phabricator.wikimedia.org/T185608#4283388 (10Legoktm) 05Open>03Resolved Yep :) [17:01:27] addshore: ok, cloning mw will take a while, forgot about that [17:22:03] zeljkof: indeeed [17:22:42] addshore: pulling containers... :D [17:22:46] forgot about that too [17:23:45] (03CR) 10Hashar: "Just make sure it passes with a 'check experimental'. At least on the master branch it should be working just fine." [integration/config] - 10https://gerrit.wikimedia.org/r/440325 (https://phabricator.wikimedia.org/T162455) (owner: 10Zfilipin) [17:24:37] zeljkof: indeedy [17:24:42] download all of the things [17:25:03] almost there... [17:25:34] `Your development environment is running` [17:28:11] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4283514 (10mmodell) I can't find a `smccandlish` or `mech` user in gerrit. Is there another name... [17:28:19] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4283517 (10hashar) [17:28:30] addshore: ok, works fine on my laptop :/ does not work on my desktop (it used to work there) [17:29:05] zeljkof: what does your localsettings look like? [17:29:08] it's too late to debug today, I'll ping you these days [17:29:15] the basic from the readme [17:29:23] is core all pulled and up to date with no local changes? and have you run composer update? [17:29:30] those are the next things :) [17:29:44] just cloned core, ran the composer command from the readme [17:31:10] hosts file also up to date [17:31:29] I can't see any differences between the machines, works on laptop, does not work on desktop [17:31:34] both macos [17:31:57] same versions of everything as far as I can see, same setup, will take a look when I'm not this tired [17:40:48] thcipriani: im pretty happy with all of the testing that has been done now [17:40:56] shall we make a branch together? [17:41:55] addshore: sure thing! [17:42:06] .666 or .999? ;) [17:42:37] let's just do 999: lamer, but less to explain later :) [17:42:54] hehe, okay ;) [17:42:59] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4283577 (10kaldari) @mmodell: Didn't fix it for me. I can log into Gerrit on the web, but my account has nothing in it. Still can't use gerrit via git, so no change for me. [17:43:13] I'd like to think we just created a new standard, when creating weird branches start at 999 and work down 998 997 996.... [17:43:14] addshore: so my plan was just to cut from master like always since that's what's running in beta: sound sane to you? [17:43:29] thcipriani: yes [17:43:48] I was thinking about this earlier, the other route is the cherry pick the patches, but that is messier, and then that code hasn't been running on beta too [17:43:55] so yes, branch from master sounds better [17:44:09] kaldari can you save in your prefence? [17:44:13] preference [17:44:18] I'll check [17:44:27] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4283580 (10mmodell) @kaldari: ok I'll poke around some more and see what else I can find. [17:44:39] addshore: kk, I'll kick off the process outlined: https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_gerrit I can let you know when that's started/done unless you want to look over my shoulder for it [17:44:51] paladox: Yes, I can save prefs [17:44:56] ok [17:44:59] * addshore will just watch on IRC :) [17:45:02] kaldari is your ssh key in settings? [17:45:11] lemme check [17:45:24] kaldari https://gerrit.wikimedia.org/r/settings/#SSHKeys [17:45:47] https://gerrit.wikimedia.org/r/dashboard/rkaldari@wikimedia.org [17:45:52] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q4 Selenium framework improvements - https://phabricator.wikimedia.org/T190994#4283586 (10zeljkofilipin) [17:45:55] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Document running Selenium tests targeting custom MediaWiki install - https://phabricator.wikimedia.org/T196561#4283585 (10zeljkofilipin) 05Open>03Resolved [17:46:10] kaldari: are you logging in with user kaldari or kaldari2? [17:46:17] you had 3 accounts, apparently [17:46:22] thcipriani: *goes to refresh himself on what that page says though* [17:46:26] kaldari, kaldari2 was a fake account set up by Ori many years ago [17:46:41] so it's safe to disable kaldari2? [17:47:05] yes [17:48:04] thcipriani: that is quite a list of points [17:48:51] addshore: indeed, long but thorough, I like to think as little as possible [17:49:06] kaldari: ok I removed your email address for kaldari2 account. Can you see any difference now? [17:50:30] thcipriani: I have my own little cheat sheet for the deploy stuff I need to remember https://wikitech.wikimedia.org/wiki/User:Addshore/Deployments [17:51:34] nice [17:52:15] twentyafterfour: weird, now I can't log in as kaldari [17:52:35] I get authentication failure [17:53:16] kaldari does it say a specific error? [17:53:55] kaldari: try one more time? [17:53:57] addshore: cutting the branch now, will be a another 10 minutes or so there, then I have to checkout and stage on deploy1001, I'll poke you when that's done. We're close to our train hour cutoff so I'll probably have to do the sync after since l10n on a new branch is currently taking about 40 minutes. Sound good to you? [17:54:03] something keeps adding back the bogus user account data [17:54:09] I can log in now [17:54:09] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4283621 (10Dzahn) I think it's still an LDAP issue just like earlier in this ticket and nothing... [17:54:21] I had to remove the duplicate email address just now, gerrit added it back automatically [17:54:22] still nothing in my account though [17:54:22] thcipriani: yup, sounds good to me [17:54:27] twentyafterfour ah [17:54:30] that is the index [17:54:37] stale index [17:54:53] paladox: why would it do that? I'm updating the account via rest api, not via git [17:55:05] the index can get stale for some reason [17:55:08] we had this in 2.13 [17:55:16] and they switched back to the index for 2.15 [17:55:33] https://gerrit-review.googlesource.com/Documentation/rest-api-accounts.html#index-account [17:55:37] twentyafterfour ^^ [17:55:38] paladox: running reindex says there is nothing to reindex [17:56:54] paladox: ok I did POST accounts/6099/index and accounts/78/index [17:57:02] ok [17:57:08] kaldari: any change to your account? still "empty?": [17:57:27] * paladox belives upstream should just stop using the index for accounts [17:57:33] I wonder if your data got re-associated with the newer account id [17:57:33] it causes so many problems. [17:57:47] twentyafterfour: now can't log in again [17:57:53] wtf [17:58:42] newer account id? [17:59:55] There seem to be 3 accounts for me(?), kaldari, Kaldari, and kaldari2 (the fake one Ori made) [18:00:12] kaldari: indeed [18:00:20] there is a problem with case sensitivity [18:00:43] it keeps enabling the Kaldari user even when I disable it and remove it's email address [18:00:46] it keeps coming back [18:01:02] the newer name is kaldari lowercase and the older one is upper-case [18:01:15] ah [18:01:23] paladox: 6099 is the newer account id with username capitalized [18:01:34] 78 is the account with lower-case kaldari [18:01:39] hmm [18:01:42] strange [18:02:17] we enabled the lower ldap thing [18:02:43] maybe this is caused by https://gerrit-documentation.storage.googleapis.com/Documentation/2.13.5/config-gerrit.html#ldap.localUsernameToLowerCase [18:03:01] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap.localUsernameToLowerCase [18:03:15] If set, it must be ensured that the local usernames for all existing accounts are converted to lower case, otherwise a user that has a local username that contains upper case characters will not be able to login anymore. [18:03:28] yeh [18:03:35] i wonder if we re run https://gerrit-review.googlesource.com/Documentation/pgm-LocalUsernamesToLowerCase.html will that work? [18:03:42] paladox: did we run the conversion? somehow we have users with mixed case [18:03:47] yeh [18:05:43] im pretty sure this is the index now. But no matter what we do it seems to reference kaldari account wrong it seems. [18:06:03] https://github.com/GerritCodeReview/gerrit/commit/8803103ebed704397da3217c4047526b02ca2819#diff-cd4cd89b4f2ea6d98c2f2c0aaaa78d7b [18:09:02] twentyafterfour we should see if re running it works. hopefully it will! [18:09:20] addshore: branch is cut https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/wmf/1.32.0-wmf.999, I'll hold off on staging on deploy1001 until closer to sync-time (only takes a minute) [18:09:31] okay! [18:09:45] so this will go out after the train group2 stuff? [18:12:20] yep, that's my plan [18:13:03] "username": "Kaldari" [18:13:05] what time will that roughly be? gonna go have food now [18:13:38] marxarelli: could you ping me when group2 is out the door and you've done the requisite fretting over it? [18:14:31] addshore: it'll hopefully be over by 20:00UTC, but it depends on whether any issues come up, but you have time. [18:14:57] thcipriani: ack [18:18:51] thcipriani: I thought you were taking group2 today on account of my appointments until 12:30 Pacific [18:19:19] I can do it but it'll be around 1300 [18:19:33] marxarelli: oh right! sorry about that! [18:19:54] :D [18:19:55] marxarelli: I'll do it, no worries, brain fart :) [18:20:01] choo choo [18:20:32] grr... the api won't let me rename userss [18:20:38] realm does not allow editing username [18:21:32] thcipriani: no prob! thanks for handling it! [18:21:41] I don't know if I have the ability to run the conversion job [18:38:35] paladox, twentyafterfour: Any idea what the chance of getting this resolved within the next 4 hours is? It's blocking me from being able to deploy some stuff. [18:40:12] kaldari: I'm not sure, I'm not very experienced with gerrit so I'm kindof over my head with this. I think I've narrowed it down to https://phabricator.wikimedia.org/T197257 [18:40:18] 10Gerrit, 10Release-Engineering-Team: gerrit: run "LocalUsernamesToLowerCase" utility on gerrir host to fix mixed case usernames in gerrit's accounts db - https://phabricator.wikimedia.org/T197257#4283727 (10mmodell) p:05Triage>03High [18:40:37] 10Gerrit, 10Release-Engineering-Team: gerrit: run "LocalUsernamesToLowerCase" utility on gerrit host to fix mixed case usernames in gerrit's accounts db - https://phabricator.wikimedia.org/T197257#4283727 (10mmodell) [18:41:22] kaldari, the problem is this: "The program will produce errors if there are accounts that have the same local username, but with different case. In this case the local username for these accounts is not converted to lower case." [18:41:39] and your account exists with both. and I can't find any way to delete one of them [18:41:52] and I can't seem to rename one either [18:43:11] it looks like SamanthaNguyen's account also has the same situation [18:43:27] hmm [18:43:39] ah [18:43:43] twentyafterfour you can delete accounts [18:43:45] using this: [18:44:03] https://gerrit-review.googlesource.com/admin/repos/plugins/account [18:44:35] twentyafterfour https://gerrit-review.googlesource.com/q/project:plugins/account [18:45:41] though not sure if a admin can do it [18:46:35] twentyafterfour https://gitenterprise.me/2018/05/25/gerrithub-io-and-gdpr/ says admin can do it [18:46:36] though [18:49:45] paladox: I don't know anything about gerrit plugins [18:49:53] ok [18:50:56] I imagine rolling back the gerrit version isn't a good option(?) [18:50:57] 10Gerrit, 10Release-Engineering-Team: gerrit: run "LocalUsernamesToLowerCase" utility on gerrit host to fix mixed case usernames in gerrit's accounts db - https://phabricator.wikimedia.org/T197257#4283788 (10mmodell) https://wiki.linaro.org/Platform/Systems/Gerrit/DupAccounts discusses how to delete dupe accou... [18:51:14] kaldari nope as it moved data into the repo [18:51:17] from the db [18:51:39] oh [18:51:44] i forgot about flush caches [18:51:51] 10Gerrit, 10Release-Engineering-Team: gerrit: run "LocalUsernamesToLowerCase" utility on gerrit host to fix mixed case usernames in gerrit's accounts db - https://phabricator.wikimedia.org/T197257#4283793 (10mmodell) from the [[ https://gerrit-review.googlesource.com/Documentation/pgm-LocalUsernamesToLowerCase... [18:52:56] twentyafterfour https://wiki.linaro.org/Platform/Systems/Gerrit/DupAccounts should be similar in All-Users [18:52:59] except you git commit [18:59:12] paladox: except I can't git commit and push because of the validation [18:59:18] oh [18:59:25] twentyafterfour how have you been fixing it? [19:00:10] i am guessing we should fix the validation? kaldari you could create a seperate gerrit account temporarily if you need to work on your thing :) [19:04:23] To ssh://twentyafterfour@gerrit.wikimedia.org:29418/All-Users [19:04:25] ! [remote rejected] HEAD -> refs/users/99/6099 (prohibited by Gerrit: ref update access denied) [19:05:04] hmm [19:05:07] oh [19:06:24] (03PS1) 10Paladox: Modify access rules [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440389 [19:06:31] twentyafterfour ^^ [19:09:08] (03CR) 1020after4: [C: 032] Modify access rules [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440389 (owner: 10Paladox) [19:09:24] (03CR) 1020after4: [V: 032 C: 032] Modify access rules [All-Users] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440389 (owner: 10Paladox) [19:10:06] 10Gerrit: No way to mark patches as not WIP (which blocks merging) - https://phabricator.wikimedia.org/T197238#4283816 (10Tgr) Sorry. It seems me and others who reported this have just missed the button on the new interface (where it replaces the "Reply" button, with the same hard-to-read color, so it's somewhat... [19:14:28] twentyafterfour did that work? :) [19:15:05] ugh [19:15:11] paladox: "External id username:kaldari cannot be deleted" [19:15:18] oh [19:15:22] meh [19:15:36] it's a freaking circle of dispair [19:15:49] yeh vary enoying too [19:16:13] could you add all this info to https://bugs.chromium.org/p/gerrit/issues/detail?id=9256 [19:16:18] or create a new issue [19:16:23] since upstream need to fix this [19:16:37] but they need the info to do it as apparently there not having this issue. [19:21:13] twentyafterfour im trying the delete plugin on https://gerrit.git.wmflabs.org/r/ [19:21:17] it includes a ssh command! [19:21:42] paladox: the real problem is kaldari's account has an uppercase username [19:21:52] yeh [19:21:52] so the dupe account will just be recreated automatically every time [19:22:01] we have to correct the username but that's the one piece I can't seem to modify [19:22:19] remove the old username instead? [19:22:30] and run the lower case cmd [19:22:46] paladox: but then kaldari will lose all associated history [19:22:57] because it's all associated with the old user id no the new one [19:23:06] twentyafterfour hmm i am going to test this, i think gerrit may do it by email [19:23:27] i have alot of fake data on here https://gerrit.git.wmflabs.org/r/ [19:23:35] using a command they provide to do the fake data [19:31:40] realm does not allow editing name [19:31:48] i guess that's why you carn't delete the user [19:31:59] paladox: yep [19:32:03] feel free to experiment with the bogus Kaldari2 account if you need to [19:32:21] kaldari thanks, though to unblock you, you should create a spare account for now :) [19:32:33] ok [19:32:39] as this could take a while, needs patching upstream most likly or upstream may have work arounds for us [19:32:44] kaldari: we can grant any privs you had on the old account [19:32:59] ack [19:33:14] paladox: do you know where notedb is stored on disk? [19:33:20] twentyafterfour yep [19:33:30] /srv/gerrit/git/All-Users.git [19:33:38] if we can directly modify the db and bypass the validation, then restart gerrit, everything should be resolved [19:33:54] ah [19:34:15] need to flush the cache too [19:34:36] yeah [19:36:46] i hope that will work [19:36:55] but i fear the index is just going to overwrite it [19:38:15] twentyafterfour: OK, I created a new account named valerie. Can you give it some privs? [19:44:51] yay, I can clone again :) [19:45:07] https://gerrit-review.googlesource.com/c/gerrit/+/92830 [19:45:18] twentyafterfour that was the fix we were using in 2.13 ^^ [19:45:23] though we doin't have the db now [19:48:39] kaldari: ok [19:48:50] adding you to some groups [19:54:30] kaldari: ok you should have the same access as before [19:54:49] thanks! seems to be working! [19:58:17] Thanks for the help and good luck getting the mess resolved. Let me know if there's anything else I can do to help. [20:10:49] 10Release-Engineering-Team, 10Multi-Content-Revisions (MCR-SDC phase 1), 10User-Addshore: Deploy some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4283964 (10Addshore) For phabricator searchability the branch name is wmf/1.32.0-wmf.999 [20:20:01] (03PS3) 10Krinkle: Archive the ClientSide extension [integration/config] - 10https://gerrit.wikimedia.org/r/439572 (https://phabricator.wikimedia.org/T196617) (owner: 10MarcoAurelio) [20:20:12] (03CR) 10Krinkle: [C: 032] Archive the ClientSide extension [integration/config] - 10https://gerrit.wikimedia.org/r/439572 (https://phabricator.wikimedia.org/T196617) (owner: 10MarcoAurelio) [20:24:25] https://github.com/GerritCodeReview/gerrit/commit/10aa4e2bbf3472f2611e5493365659b9fd8fe9e9#diff-5513a6cc9dc99f790a2551c5633fd87f [20:27:43] thcipriani: should we deploy the fix for https://phabricator.wikimedia.org/T197262 immediately? [20:29:28] MaxSem: I can deploy it after the current deployment (running l10nupdate, may take a few). What is the user impact? I can stop sync and deploy if need be. [20:30:03] it's not terribly spammy in the logs, not blocking everything else out, just at the top of the list [20:30:37] hmm [20:30:42] it seems it's not using the cache [20:30:44] https://github.com/GerritCodeReview/gerrit/commit/38b17c63dfa7257467fb52dba47388ebab07d0ea#diff-5513a6cc9dc99f790a2551c5633fd87f [20:30:48] index i mean [20:30:52] twentyafterfour ^^ [20:31:01] and that change was in 2.14 [20:31:23] 124 fatals in the last 15 minutes - but because it's through the API, it's less visible to users so I say it can wait [20:32:10] MaxSem: ok, thank you for the quick work! I will backport and deploy after this scap run [20:33:50] *reads up* [20:34:38] I guess that should be backported to .8 and .999 [20:36:56] twentyafterfour did editing in /srv/gerrit/git/All-Users.git work? [20:48:01] 10Phabricator, 10Patch-For-Review: Phabricator footer links should use Special:MyLanguage for links that support it - https://phabricator.wikimedia.org/T196836#4284098 (10Framawiki) 05Open>03Resolved [21:47:35] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/439572/ [21:47:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:03:16] PROBLEM - SSH on integration-slave-docker-1016 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:05] RECOVERY - SSH on integration-slave-docker-1016 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [23:16:34] 10Release-Engineering-Team: Scap error from mwdebug2001.codfw.wmnet: sync: write failed on "/srv/mediawiki/wmf-config/InitialiseSettings.php": No space left on device (28) - https://phabricator.wikimedia.org/T197275#4284384 (10Jdforrester-WMF) [23:17:42] 10Release-Engineering-Team, 10Operations: Scap error from mwdebug2001.codfw.wmnet: sync: write failed on "/srv/mediawiki/wmf-config/InitialiseSettings.php": No space left on device (28) - https://phabricator.wikimedia.org/T197275#4284395 (10Krenair) [23:48:24] 10Release-Engineering-Team, 10Operations: Scap error from mwdebug2001.codfw.wmnet: sync: write failed on "/srv/mediawiki/wmf-config/InitialiseSettings.php": No space left on device (28) - https://phabricator.wikimedia.org/T197275#4284384 (10thcipriani) I pruned 1.32.0-wmf.{1,2,3} to clear out some space, but s... [23:50:35] 10Release-Engineering-Team, 10Operations: Scap error from mwdebug2001.codfw.wmnet: sync: write failed on "/srv/mediawiki/wmf-config/InitialiseSettings.php": No space left on device (28) - https://phabricator.wikimedia.org/T197275#4284384 (10Reedy) /dev/vda1 on mwdebug2002 is 39GB. mwdebug1001 has 49GB [23:59:23] 10Continuous-Integration-Config, 10RelEng-Archive-FY201718-Q2, 10Composer, 10Upstream, 10Wikimedia-log-errors (Shared Build Failure): Composer\Downloader\TransportException in jobs with mediawiki or composer - https://phabricator.wikimedia.org/T182266#4284446 (10Krinkle)