[00:03:09] Hmm. Navigating to https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/439311/ loads the content, but then says "The page you requested was not found, or you do not have permission to view this page.". Known bug? [00:04:55] https://gerrit.wikimedia.org/r/changes/mediawiki%2Fcore~REL1_31~I79f1d28b54532a7495fb8e205c9b6636016587d7/revisions/2/zuul~crd [00:05:27] James_F nope not a known but [00:05:33] https://gerrit.wikimedia.org/r/changes/mediawiki%2Fcore~REL1_31~I79f1d28b54532a7495fb8e205c9b6636016587d7/revisions/2/zuul~crd returns the error though [00:05:40] Multiple changes found for mediawiki%2Fcore~REL1_31~I79f1d28b54532a7495fb8e205c9b6636016587d7 [01:06:06] (03CR) 10Legoktm: [C: 032] docker: Switch npm-php image over to PHP 7 [integration/config] - 10https://gerrit.wikimedia.org/r/440027 (https://phabricator.wikimedia.org/T196956) (owner: 10Legoktm) [01:08:12] (03Merged) 10jenkins-bot: docker: Switch npm-php image over to PHP 7 [integration/config] - 10https://gerrit.wikimedia.org/r/440027 (https://phabricator.wikimedia.org/T196956) (owner: 10Legoktm) [01:10:25] !log deploying https://gerrit.wikimedia.org/r/440027 [01:10:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:30:44] (03PS1) 10Legoktm: Bump mediawiki-phpcs-dryrun image [integration/config] - 10https://gerrit.wikimedia.org/r/440040 [01:33:13] (03CR) 10Legoktm: [C: 032] Bump mediawiki-phpcs-dryrun image [integration/config] - 10https://gerrit.wikimedia.org/r/440040 (owner: 10Legoktm) [01:34:27] (03PS1) 10Legoktm: Use npm-php:0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/440042 (https://phabricator.wikimedia.org/T196956) [01:35:20] (03Merged) 10jenkins-bot: Bump mediawiki-phpcs-dryrun image [integration/config] - 10https://gerrit.wikimedia.org/r/440040 (owner: 10Legoktm) [01:38:00] (03CR) 10Legoktm: [C: 032] Use npm-php:0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/440042 (https://phabricator.wikimedia.org/T196956) (owner: 10Legoktm) [01:40:14] (03Merged) 10jenkins-bot: Use npm-php:0.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/440042 (https://phabricator.wikimedia.org/T196956) (owner: 10Legoktm) [01:43:20] 10Continuous-Integration-Config, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking), 10Readers-Web-Kanbanana-Board: mwext-MobileFrontend-npm-run-lint-modules-docker failing - node script running as php - https://phabricator.wikimedia.org/T196956#4277695 (10Legoktm) 05Open>03Resolved a:03Legoktm Th... [02:15:48] PROBLEM - Host deployment-redis02 is DOWN: CRITICAL - Host Unreachable (10.68.16.231) [02:15:50] PROBLEM - Host deployment-redis01 is DOWN: CRITICAL - Host Unreachable (10.68.16.177) [02:18:01] PROBLEM - Host deployment-dumps-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.21.153) [02:24:20] PROBLEM - Host deployment-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (10.68.21.200) [02:36:35] Project beta-scap-eqiad build #211575: 04FAILURE in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/211575/ [02:48:01] Yippee, build fixed! [02:48:01] Project beta-scap-eqiad build #211576: 09FIXED in 4 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/211576/ [02:57:13] I can't seem to do a git pull from gerrit. Anyone else having issues? [03:14:48] kaldari: works for me... [03:15:06] kaldari: can you ping gerrit? [03:15:20] weird. when I go to https://gerrit.wikimedia.org/r/#/dashboard/self, it's totally empty, but shows the headers. [03:15:49] Yeah, I can ping it... [03:16:14] When I try to git pull it gives me an ssh authentication error. [03:16:28] like my gerrit account has been reset or something [03:16:47] I can still ssh to terbium, etc. [03:20:33] Gerrit accounts have different ssh keys than prod... [03:31:22] Hello! Someone please look over T196219 :D [03:31:22] T196219: Enable ULS webfonts by default at Burmese Wikipedia (mywiki) - https://phabricator.wikimedia.org/T196219 [04:28:00] (03PS1) 10Prtksxna: Add tests for oojs/router [integration/config] - 10https://gerrit.wikimedia.org/r/440052 (https://phabricator.wikimedia.org/T191374) [04:29:29] (03CR) 10jerkins-bot: [V: 04-1] Add tests for oojs/router [integration/config] - 10https://gerrit.wikimedia.org/r/440052 (https://phabricator.wikimedia.org/T191374) (owner: 10Prtksxna) [04:30:06] 10Continuous-Integration-Config, 10OOjs-Router, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Move OOjs Router to Gerrit (from Diffusion) - https://phabricator.wikimedia.org/T191374#4277782 (10Prtksxna) Thanks @MarcoAurelio *** >>! In T191374#4274786, @MarcoAurelio wrote: > Okay so I'l... [04:32:05] (03PS2) 10Prtksxna: Add tests for oojs/router [integration/config] - 10https://gerrit.wikimedia.org/r/440052 (https://phabricator.wikimedia.org/T191374) [04:42:01] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:07:03] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4277822 (10Marostegui) >>! In T196840#4277313, @mmodell wrote: > @marostegui: I canceled some of the queued jobs which should have helped somewhat. The only thing I know to do... [05:09:52] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4277823 (10mmodell) I've got the queue down to 3.1M by canceling jobs. There is still write traffic involved even to delete the jobs so it hasn't really reduced the traffic as... [05:17:01] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:39:54] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4277839 (10jcrespo) p:05High>03Normal I don't think this is high from our perspective- they have dedicated db resources and the replica is up to data, and were aware of the... [05:43:44] (03PS1) 10Legoktm: Configure jobs for oojs/router [integration/config] - 10https://gerrit.wikimedia.org/r/440054 (https://phabricator.wikimedia.org/T191374) [05:44:55] (03CR) 10Prtksxna: "I'll abandon I56665c19c03de7a89234fc1af68c58fe54dc04be?" [integration/config] - 10https://gerrit.wikimedia.org/r/440054 (https://phabricator.wikimedia.org/T191374) (owner: 10Legoktm) [05:47:00] (03CR) 10Legoktm: [C: 032] Configure jobs for oojs/router [integration/config] - 10https://gerrit.wikimedia.org/r/440054 (https://phabricator.wikimedia.org/T191374) (owner: 10Legoktm) [05:47:24] (03Abandoned) 10Legoktm: Add tests for oojs/router [integration/config] - 10https://gerrit.wikimedia.org/r/440052 (https://phabricator.wikimedia.org/T191374) (owner: 10Prtksxna) [05:49:14] (03Merged) 10jenkins-bot: Configure jobs for oojs/router [integration/config] - 10https://gerrit.wikimedia.org/r/440054 (https://phabricator.wikimedia.org/T191374) (owner: 10Legoktm) [05:55:00] !log deployed https://gerrit.wikimedia.org/r/440054 [05:55:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:01:04] (03Abandoned) 10Chad: WIP: Bazel build with java8 [integration/config] - 10https://gerrit.wikimedia.org/r/434006 (owner: 10Chad) [06:01:39] 10Continuous-Integration-Config, 10OOjs-Router, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Move OOjs Router to Gerrit (from Diffusion) - https://phabricator.wikimedia.org/T191374#4277846 (10Legoktm) [06:02:32] 10Continuous-Integration-Config, 10OOjs-Router, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Move OOjs Router to Gerrit (from Diffusion) - https://phabricator.wikimedia.org/T191374#4103087 (10Legoktm) >>! In T191374#4277782, @Prtksxna wrote: > Just out of curiosity — will it be Gerrit >... [06:08:56] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4277853 (10mmodell) The gerrit notedb migration was a one time event, so it shouldn't really be something that happens with every update. [06:26:48] 10Continuous-Integration-Config, 10OOjs-Router, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Move OOjs Router to Gerrit (from Diffusion) - https://phabricator.wikimedia.org/T191374#4277858 (10Prtksxna) Thanks @Legoktm! [06:29:45] kaldari: hi, did you try upper and lower case? [06:44:45] (03PS1) 10Legoktm: Revert "Allow project owners to submit" [oojs/router] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440058 [06:44:52] (03CR) 10Legoktm: [V: 032 C: 032] Revert "Allow project owners to submit" [oojs/router] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440058 (owner: 10Legoktm) [06:46:25] (03PS1) 10Legoktm: doc: Add OOjs Router [integration/docroot] - 10https://gerrit.wikimedia.org/r/440059 [06:46:41] (03CR) 10Legoktm: [C: 032] doc: Add OOjs Router [integration/docroot] - 10https://gerrit.wikimedia.org/r/440059 (owner: 10Legoktm) [06:47:16] (03Merged) 10jenkins-bot: doc: Add OOjs Router [integration/docroot] - 10https://gerrit.wikimedia.org/r/440059 (owner: 10Legoktm) [06:47:22] (03CR) 10jenkins-bot: doc: Add OOjs Router [integration/docroot] - 10https://gerrit.wikimedia.org/r/440059 (owner: 10Legoktm) [07:04:20] 10Continuous-Integration-Config, 10OOjs-Router, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Move OOjs Router code feview from Differential to Gerrit - https://phabricator.wikimedia.org/T191374#4277939 (10hashar) [07:05:02] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4277942 (10hashar) [07:19:35] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Build base image for math extension pipeline tests - https://phabricator.wikimedia.org/T196939#4273473 (10hashar) A summary of our chat yesterday: You can look at the releng/quibble-stretch container for some inspiration. * you would need composer to i... [07:21:54] !log github: deleting archived repo wikimedia/operations-software-tessera | TT186096 [07:21:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:23:51] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4277978 (10hashar) [07:24:38] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4096177 (10Dzahn) >>! In T191182#4096246, @HappyDog wrote: > If you mean internal WMF dev-ops stuff that only WMF employees care about, then perhaps that matters less No... [07:26:04] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-05 (1.32.0-wmf.7)), and 2 others: Wikimedia\Rdbms\ChronologyProtector::initPositions: expected but failed to find positio... - https://phabricator.wikimedia.org/T194403#4277983 [07:32:06] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4277998 (10TerraCodes) >>! In T191182#4111915, @Aklapper wrote: > I explained in T191182#4103647 why I think this is wrong. So I am reopening this task. If Differential h... [07:44:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikimedia-log-errors (Jenkins Failure): selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4052385 (10hashar) >>! In T189762#4273468, @Smalyshev wr... [07:45:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikimedia-log-errors (Jenkins Failure): selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4278028 (10hashar) [07:45:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Lexicographical data, 10Math, and 3 others: MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4109191 (10hashar) [07:46:07] zeljkof: hello! There is also a task about Wikibase selenium tests being flappy [07:46:34] zeljkof: and there are at least two occurences on that tasks that I strongly link to the cookie/session issue for the Math/Wikibase tests [08:01:33] hashar: does that mean the problems will be resolved with the patch? [08:06:09] Some how gerrit created two accounts for kaldari [08:06:25] Which should not be possible as you cannot use the same email for the same account [08:07:44] I suspect that it would have also created another external Id [08:16:38] zeljkof: probably yes [08:16:56] zeljkof: at least the problem of user being mysteriously logged out will be solved [08:17:43] 10Continuous-Integration-Config, 10OOjs-Router, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Move OOjs Router code review from Differential to Gerrit - https://phabricator.wikimedia.org/T191374#4278110 (10Aklapper) [08:17:46] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4278111 (10EddieGP) >>! In T191182#4277998, @TerraCodes wrote: >>>! In T191182#4111915, @Aklapper wrote: >> I explained in T191182#4103647 why I think this is wrong. So I... [08:18:06] paladox: we noticed those dupe accounts yesterday as well. Another example is searching for: owner:jbranaa that shows two accounts sharing the same mail [08:19:16] paladox: maybe that is due to ldap and case sensitiveness. We would have to look at the database fields I guess [08:19:22] hashar: that sounds like a bug [08:19:30] hashar: it’s not in the db now :) [08:19:33] It’s in notedb [08:19:38] All-Users [08:19:42] Is the repo [08:19:53] \o/ [08:20:06] You would need to clone and also you would have to grant your self the db right in gerrit to see all the references [08:21:10] Ah you should have the right [08:21:14] As it’s for admins :) [08:22:08] (03PS1) 10Paladox: Modify access rules [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440073 [08:22:12] hashar: ^^ [08:23:07] paladox: that drops the "security" group apparently https://gerrit.wikimedia.org/r/c/All-Projects/+/440073/1/groups [08:23:24] Oh [08:23:32] hashar: does that group exist? [08:23:46] no idea :] [08:24:13] security is not visible to all [08:24:20] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-05 (1.32.0-wmf.7)), and 2 others: Wikimedia\Rdbms\ChronologyProtector::initPositions: expected but failed to find positio... - https://phabricator.wikimedia.org/T194403#4278157 [08:24:34] and it has members [08:24:42] so I guess you do not have access to it and gerrit drop it [08:24:49] when you craft the change [08:24:52] which would be a bug :] [08:25:27] Heh [08:25:29] Ah [08:25:35] Ok yeh that would be a bug [08:25:57] (03PS2) 10Paladox: Modify access rules [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440073 [08:26:04] hashar: done [08:28:05] paladox: mind filling a bug about the dupe accounts? [08:28:13] I will not be able to look at it today, but maybe tomorrow [08:28:33] Yeh I can later (currently in a class) :) [08:28:51] paladox: oh. Focus on your class!! that is important! [08:29:02] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Stop using Differential for code review - https://phabricator.wikimedia.org/T191182#4278166 (10HappyDog) [08:29:09] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Stop using Differential for code review - https://phabricator.wikimedia.org/T191182#4096177 (10HappyDog) >>! In T191182#4278111, @EddieGP wrote: > The proposal here is to //stop using// differential. This has come up a couple of times - I've upd... [08:53:27] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-05 (1.32.0-wmf.7)), and 2 others: Wikimedia\Rdbms\ChronologyProtector::initPositions: expected but failed to find positio... - https://phabricator.wikimedia.org/T194403#4278201 [08:59:11] 10Release-Engineering-Team (Kanban), 10User-greg: Figure out how RelEng can better communicate accomplishments - https://phabricator.wikimedia.org/T197050#4277481 (10zeljkofilipin) We could write blog posts about things we do. Some posts are already there at our team blog. I //**love**// lightning talks (5 min... [09:04:41] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Database, 10Performance-Team, 10MW-1.32-release-notes (WMF-deploy-2018-06-05 (1.32.0-wmf.7)), and 2 others: Wikimedia\Rdbms\ChronologyProtector::initPositions: expected but failed to find positio... - https://phabricator.wikimedia.org/T194403#4278279 [09:07:25] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:38:24] 10MediaWiki-Codesniffer: remove [optional] from parameter docs - https://phabricator.wikimedia.org/T196773#4268031 (10thiemowmde) Personally I fully agree with what is briefly outlined in the task description: If a parameter is optional or not is always visible from the function header in PHP (in contrast to Jav... [09:59:09] hashar: heh ok, I can file it now [10:01:13] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4278455 (10Paladox) [10:01:21] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4278465 (10Paladox) p:05Triage>03High [10:01:57] hashar done ^^ [10:11:57] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4278455 (10Dzahn) T138672 might be an example of this bug [10:12:01] also see T138672 [10:12:01] T138672: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672 [10:12:44] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#2406941 (10Dzahn) This might be the Gerrit duplicate user issue: T197083 [10:14:53] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4278524 (10Paladox) When searching for that user it couldn’t find them at least in the search bar. [10:16:39] mutante: ^^ [10:16:51] I think it’s a different issue [10:17:03] As it was happening before we updated [10:17:23] 10Gerrit, 10Operations, 10Traffic, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4096194 (10Dzahn) gerrit.wmfusercontent.org now exists in cache::misc and requests would be forwarded to cobalt as the backend. This unblocked this to a certain extent because avatar... [10:17:30] 10Gerrit, 10Operations, 10Traffic, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4278528 (10Dzahn) p:05Triage>03Normal [10:19:56] paladox: i am not sure that is actually confirmed since "it" was 2 different things. First they had an issue with Phab and LDAP .. then Andrew did stuff about it.. then the user says they have duplicate Gerrit user. Andrew says now it's a seaparate issue [10:20:03] it can be the same user with 2 bugs [10:20:28] Gerrit could have been broken before or after or during the Phab thing? [10:22:02] paladox: what about those changes in the past regarding auth .. was any of that merged with a comment like "doesnt affect anything until 2.15" [10:32:31] (03PS1) 10Hashar: Migrate GoogleLogin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440089 (https://phabricator.wikimedia.org/T183512) [10:33:23] (03CR) 10Hashar: [C: 032] Migrate GoogleLogin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440089 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:35:30] (03Merged) 10jenkins-bot: Migrate GoogleLogin to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440089 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:36:13] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4278620 (10hashar) [10:37:48] 10Continuous-Integration-Config, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking), 10Readers-Web-Kanbanana-Board: mwext-MobileFrontend-npm-run-lint-modules-docker failing - node script running as php - https://phabricator.wikimedia.org/T196956#4278622 (10Jhernandez) Thanks for the fix. We’ll have to f... [10:48:45] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Stop using Differential for code review - https://phabricator.wikimedia.org/T191182#4278655 (10Aklapper) The monthly Phab summary emails on wikitech-l@ show that we've had {20,23,19,22,19,28,29,30,19,26,24,24} Differential users in the last 12 mo... [10:52:09] 10Gerrit, 10Developer-Relations, 10Google-Code-in-2018, 10Documentation: Gerrit's test instance gerrit.git.wmflabs.org is not quite visible in the docs; no clear instructions how to use it - https://phabricator.wikimedia.org/T193788#4278675 (10Aklapper) [11:17:55] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4278788 (10hashar) [11:18:50] 10Continuous-Integration-Config, 10OOjs-Router, 10Repository-Admins, 10Patch-For-Review, 10User-MarcoAurelio: Move OOjs Router code review from Differential to Gerrit - https://phabricator.wikimedia.org/T191374#4278792 (10MarcoAurelio) Yep @Prtksxna Whatever you commit/merge on Gerrit will be mirrored on... [11:23:48] (03PS1) 10Hashar: Switch Mpdf to use composer [integration/config] - 10https://gerrit.wikimedia.org/r/440094 (https://phabricator.wikimedia.org/T188523) [11:24:09] (03CR) 10Hashar: [C: 032] Switch Mpdf to use composer [integration/config] - 10https://gerrit.wikimedia.org/r/440094 (https://phabricator.wikimedia.org/T188523) (owner: 10Hashar) [11:26:15] (03Merged) 10jenkins-bot: Switch Mpdf to use composer [integration/config] - 10https://gerrit.wikimedia.org/r/440094 (https://phabricator.wikimedia.org/T188523) (owner: 10Hashar) [11:34:53] 10Release-Engineering-Team, 10Multi-Content-Revisions (MCR-SDC phase 1), 10User-Addshore: Investigate possibility of having some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4278840 (10Addshore) The patches will be: * PageUpdater: https://gerrit.wik... [11:35:14] 10Release-Engineering-Team, 10Multi-Content-Revisions (MCR-SDC phase 1), 10User-Addshore: Investigate possibility of having some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4278841 (10Addshore) [11:42:36] 10Release-Engineering-Team, 10Multi-Content-Revisions (MCR-SDC phase 1), 10User-Addshore: Deploy some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4278851 (10Addshore) [11:47:53] mutante: it’s trying to create a duplicate user [11:47:58] But the check is working :) [11:48:15] Some how when notedb did it’s migration things the checks did not quick in [11:48:23] Thus we have duplicate accounts now [12:07:10] hashar: we should also try to fix https://phabricator.wikimedia.org/T138672 at some point [12:07:16] I think it may be a duplicate external if [12:07:17] Id [12:14:27] (03PS1) 10Hashar: Migrate Mpdf to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440103 (https://phabricator.wikimedia.org/T188523) [12:14:48] (03CR) 10Hashar: [C: 032] Migrate Mpdf to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440103 (https://phabricator.wikimedia.org/T188523) (owner: 10Hashar) [12:15:07] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4278885 (10hashar) [12:15:29] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4249240 (10hashar) [12:17:26] (03Merged) 10jenkins-bot: Migrate Mpdf to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440103 (https://phabricator.wikimedia.org/T188523) (owner: 10Hashar) [12:38:14] (03PS1) 10MarcoAurelio: Archive the CustomPage skin [skins/CustomPage] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440110 (https://phabricator.wikimedia.org/T196429) [12:38:56] (03PS2) 10MarcoAurelio: Archive the CustomPage skin [skins/CustomPage] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440110 (https://phabricator.wikimedia.org/T196429) [12:53:59] (03PS1) 10Hashar: Migrate Wikisource to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440111 (https://phabricator.wikimedia.org/T183512) [12:56:50] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4278974 (10hashar) [12:57:11] (03CR) 10Hashar: [C: 032] Migrate Wikisource to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440111 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [12:58:39] (03Merged) 10jenkins-bot: Migrate Wikisource to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/440111 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [13:18:46] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:20:14] PROBLEM - SSH on integration-slave-docker-1016 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:04] RECOVERY - SSH on integration-slave-docker-1016 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [13:32:46] I can't suddenly make patches in gerrit, it tells me my public key is not valid [13:33:39] is it down or something is wrong on my side [13:40:29] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4279142 (10Liuxinyu970226) @tosfos The [[https://www.mediawiki.org/wik... [13:48:16] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4279205 (10hashar) >>! In T183512#4279142, @Liuxinyu970226 wrote: > @H... [13:48:29] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4279211 (10hashar) [13:49:08] (03PS1) 10Hashar: Archive CustomPage extension [integration/config] - 10https://gerrit.wikimedia.org/r/440125 (https://phabricator.wikimedia.org/T197102) [13:49:31] (03CR) 10Hashar: [C: 032] Archive CustomPage extension [integration/config] - 10https://gerrit.wikimedia.org/r/440125 (https://phabricator.wikimedia.org/T197102) (owner: 10Hashar) [13:50:56] (03Merged) 10jenkins-bot: Archive CustomPage extension [integration/config] - 10https://gerrit.wikimedia.org/r/440125 (https://phabricator.wikimedia.org/T197102) (owner: 10Hashar) [13:51:59] hashar: where's the coffee-script npm test defined? zuul complains that pack is deprected and should be coffeescript instead [13:53:21] 10Continuous-Integration-Infrastructure, 10AntiSpoof, 10Patch-For-Review: AntiSpoof extension does not pass quibble-vendor-mysql-php70-docker - https://phabricator.wikimedia.org/T195020#4279232 (10hashar) MariaDB has been changed in the Quibble containers to use `binary`. 7d740b5167e1625c690d6f59b6b0e0ac35c4... [13:54:12] 10Continuous-Integration-Infrastructure, 10AntiSpoof, 10Patch-For-Review: AntiSpoof extension does not pass quibble-vendor-mysql-php70-docker - https://phabricator.wikimedia.org/T195020#4279235 (10hashar) I have triggered the Quibble jobs against the dummy change https://gerrit.wikimedia.org/r/c/mediawiki/ex... [14:31:13] paladox: i think i didnt get the context. which check is working? [14:41:33] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-PageCuration, 10Collaboration-Team-Triage (Collab-Team-Next-Quarter): [betalabs]: Page triage: "Uncaught TypeError: Cannot read property 'getLogPageTitle' of undefined" for 'Redirects for discussion' - https://phabricator.wikimedia.org/T196954#4279438 (10J... [15:05:52] mutante: in gerrit it checks to make sure a duplicate account is not created [15:06:06] Apparently that check failed for db -> notedb migration [15:06:14] It should work now the check [15:06:38] Amir1: try upper or lower case [15:08:46] paladox: gotcha, thanks [15:18:23] Ssh is case sensitive in gerrit [15:18:31] Compared to logging in through the ui [15:18:37] Amir1: ^^ [15:18:41] mutante: ok :) [15:19:13] let me check [15:47:39] (03PS1) 10WMDE-leszek: Added a job for analytics-wmde-toolkit-analyzer [integration/config] - 10https://gerrit.wikimedia.org/r/440149 [15:48:21] (03CR) 10jerkins-bot: [V: 04-1] Added a job for analytics-wmde-toolkit-analyzer [integration/config] - 10https://gerrit.wikimedia.org/r/440149 (owner: 10WMDE-leszek) [15:51:22] (03PS2) 10WMDE-leszek: Added a job for analytics-wmde-toolkit-analyzer [integration/config] - 10https://gerrit.wikimedia.org/r/440149 [16:04:09] so the polygerrit ui [16:04:39] I don't see it notifying me when a new version of a patch is available, when I have a tab open [16:04:53] is there something I need to tweak? or is that not a thing any more/ [16:04:56] ? [16:07:37] apergos i think it's supported from 2.15 [16:07:47] though it's possible it may only be supported from 2.16 / 3.0 [16:07:59] which one are we getting with the switchover? [16:08:06] apergos 2.15 [16:08:12] so maye [16:08:17] ok, I'll wait and see [16:08:51] ok [16:09:37] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:10:11] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.32.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T191053#4279976 (10thcipriani) 05Open>03Resolved [16:48:57] does anyone know if anything was done with logstash? starting from 16:18utc for all node services it suddenly started showing the whole json-serialized log entry as the message instead of just the actual message? [16:49:05] if not I'll just file a task.. [16:50:23] hashar ah i figured out how to use refs/meta/external-id now :) [16:50:26] you use [16:50:30] grep -rnw './' -e '' [16:50:45] as it stores it in folders now that will find the file that has the users external id [16:51:42] you git clone All-Users as your ssh user (which has admin and you need to grant the access db to the admins) then you edit .git/config and replace refs/heads with refs/ then git pull [16:52:00] and then git checkout origin/meta/external-ids [16:52:59] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4280194 (10Paladox) [17:51:42] you git clone All-Users as your ssh user (which has admin and you need to grant the access db to the admins) then you edit .git/config... [16:58:09] Pchelolo: best to task in -operations, we don't maintain logstash :) [16:58:15] s/task/ask/ [16:58:59] greg-g: it's always the question where to ask about it :) [16:59:11] tgr did you set your primary email in gerrit? [16:59:11] yep, hot potato [16:59:13] :) [16:59:14] i get a error [16:59:19] https://gerrit.wikimedia.org/r/q/owner:%2522Gerg%25C5%2591+Tisza+%253Cgtisza%2540wikimedia.org%253E%2522 [16:59:26] Server error: Not found: gtisza@wikimedia.org [17:00:20] paladox: I don't think I can? [17:00:26] that's manager by LDAP [17:00:31] tgr oh, you can [17:00:38] through https://gerrit.wikimedia.org/r/settings/ [17:00:46] https://gerrit.wikimedia.org/r/settings/#EmailAddresses [17:01:30] the email field is not editable [17:01:44] it's set to gtisza@wikimedia.org though [17:02:06] tgr is Preferred ticked for that email? [17:02:28] I don't see any option like that [17:02:50] 10Release-Engineering-Team (Kanban), 10MW-1.31-release: Release MW 1.31 - https://phabricator.wikimedia.org/T191088#4280239 (10thcipriani) a:05thcipriani>03demon Prematurely reassigned :) [17:02:59] tgr oh you doin't see https://phabricator.wikimedia.org/F22194303 ? [17:03:02] ah, OK, there is a separate email section [17:03:12] yes, it's marked as preferred [17:03:43] ok [17:03:44] thanks [17:03:59] tgr im seeing if that error is fixed by https://gerrit-review.googlesource.com/c/gerrit/+/183410 [17:06:20] hey greg, my gerrit account got nuked somehow. Who's good to ask about that? [17:06:27] oops, greg-g [17:06:36] kaldari we have a task [17:06:38] uhhhh [17:06:41] oh good :) [17:06:49] https://phabricator.wikimedia.org/T197083 [17:06:53] thanks! [17:06:54] it hasen't nuked it [17:06:58] just it duplicated it! [17:06:59] yay [17:08:32] kaldari does ssh work? wondering if you were to clone https://gerrit.wikimedia.org/r/admin/projects/All-Users would it show the duplicate external ids [17:08:41] nope [17:09:01] can't git pull, git push, git anything [17:09:48] ah [17:09:59] it's created two external ids for you kaldari! [17:10:01] just checked [17:10:04] through rest api [17:10:09] http://gerrit.wikimedia.org/r/accounts/?q=name:Kaldari+email:rkaldari@wikimedia.org&n=2 [17:10:14] "_account_id": 78 [17:10:15] and [17:10:21] "_account_id": 6099 [17:10:30] kaldari i suppose the 78 one is your real account [17:11:14] just need to nuke 6099 i think [17:12:39] thanks [17:14:10] 10Beta-Cluster-Infrastructure, 10ChangeProp, 10Services (done): Puppet failure on deployment-cpjobqueue - https://phabricator.wikimedia.org/T196829#4280302 (10mobrovac) 05Open>03Resolved p:05Triage>03Normal a:03mobrovac I added the hiera variable as per @Joe's suggestion and now all is good. [17:14:49] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4280308 (10Paladox) @kaldari account has two external ids if i look at http://gerrit.wikimedia.org/r/accounts/?q=name:Kaldari+email:rkaldari@wikimedia.org&n=2 ``` [ { "_... [17:21:17] 10Scap, 10Cassandra, 10Maps-Sprint, 10Operations: cassandra/metrics-collector does not deploy with scap on a new install - https://phabricator.wikimedia.org/T197159#4280346 (10Gehel) [17:24:13] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:28] Anybody who can SWAT? I'd like to see at least one of my patches deployed [17:37:51] apparently what happened to kaldari shoulden't even be possible lol [17:37:56] https://groups.google.com/forum/#!topic/repo-discuss/GaV0V6zJerU [17:47:15] 10Scap, 10Cassandra, 10Maps-Sprint, 10Operations: cassandra/metrics-collector does not deploy with scap on a new install - https://phabricator.wikimedia.org/T197159#4280477 (10Gehel) Editing `/srv/deployment/cassandra/metrics-collector-cache/.config` to replace the reference to `tin` with a ref to `deploy1... [17:53:17] 10Beta-Cluster-Infrastructure, 10Services: Puppet failure on deployment-cassandra3-0[12] - https://phabricator.wikimedia.org/T196830#4280491 (10Krenair) 05Open>03Resolved Looks like it has now appeared: ```krenair@deployment-cassandra3-01:~$ apt-cache policy cassandra cassandra: Installed: 3.11.2 Candi... [18:01:10] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [18:16:00] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:22:41] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4280586 (10kaldari) FYI, I basically can't do any development work in the meantime since I can't use gerrit (can't git pull, git push, etc.). Would appreciate if this gets fixe... [18:23:14] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4280607 (10Paladox) p:05High>03Unbreak! [18:25:07] 10Beta-Cluster-Infrastructure, 10Scap: 'scap update-interwiki-cache' does not exist on deployment-tin - https://phabricator.wikimedia.org/T197166#4280669 (10MarcoAurelio) [18:27:50] paladox: how do we resolve this? [18:27:57] paladox: the kal.dari issue [18:28:10] greg-g remove the extra external id i think [18:28:17] i think this is the same problem we had a few years ago [18:28:37] https://phabricator.wikimedia.org/T197083#4280194 [18:29:03] needs https://gerrit.wikimedia.org/r/c/All-Projects/+/440073 too [18:30:35] greg-g the external id to remove is: 6099 [18:30:41] (03CR) 10Greg Grossmeier: [C: 031] "We need this so we can fix Kaldari's duplicate account issue." [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440073 (owner: 10Paladox) [18:30:53] I don't have gerrit admin privs, sadly [18:30:58] oh [18:33:09] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561#4280811 (10Krenair) Does someone already working on this want to replace the instance or shall I start a deployment-... [18:33:14] greg-g i've left instructions on how to do this (for anyone who will work on the task) :) [18:34:53] paladox: where's the list of who's a gerrit admin? I should know this but I can't remember... [18:35:04] * paladox looks [18:35:14] all of ldap/ops are [18:35:29] and [18:35:29] https://gerrit.wikimedia.org/r/admin/groups/1,members [18:35:32] greg-g ^^ [18:41:02] RECOVERY - Puppet errors on deployment-kafka-jumbo-1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:42:09] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4278455 (10mmodell) @kaldari: I'm on it [18:43:08] (03CR) 1020after4: [C: 032] Modify access rules [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440073 (owner: 10Paladox) [18:43:41] (03CR) 1020after4: [V: 032 C: 032] Modify access rules [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/440073 (owner: 10Paladox) [18:44:33] 10Beta-Cluster-Infrastructure, 10Scap: 'scap update-interwiki-cache' does not exist on deployment-tin - https://phabricator.wikimedia.org/T197166#4280921 (10Krenair) 05Open>03Invalid It lives at deployment-tin:/srv/mediawiki-staging/scap/plugins/updateinterwikicache.py so I think you just need to cd into /... [18:45:37] 10Beta-Cluster-Infrastructure, 10Scap: 'scap update-interwiki-cache' does not exist on deployment-tin - https://phabricator.wikimedia.org/T197166#4280923 (10Krenair) It's probably worth noting that it only handles interwiki.php, not interwiki-labs.php though. And would still need someone to merge the patch to... [18:51:24] grep -rnw './' -e 'Kaldari' [18:51:31] twentyafterfour ^^ that should find him in the repo [18:53:14] 10Scap: update-interwiki-cache failed: 'Namespace' object has no attribute 'force' - https://phabricator.wikimedia.org/T196642#4264272 (10Krenair) I think this is because UpdateInterwikiCache tries to use super().main but SyncFile.main has decorators that set up argparse things, which aren't get... [18:54:14] https://gerrit-review.googlesource.com/Documentation/config-accounts.html#external-ids [18:55:15] or [18:55:32] grep -rnw './' -e '6099' [18:56:00] 10Scap: update-interwiki-cache failed: 'Namespace' object has no attribute 'force' - https://phabricator.wikimedia.org/T196642#4280970 (10Krenair) But I don't think we actually want them all, e.g. we don't want `file` for this. And `message` is getting set by the subclass. Easiest fix may be to... [18:58:30] 10Scap: `scap update-interwiki-cache` broken - https://phabricator.wikimedia.org/T192469#4280973 (10Krenair) [18:58:32] 10Scap: update-interwiki-cache failed: 'Namespace' object has no attribute 'force' - https://phabricator.wikimedia.org/T196642#4280975 (10Krenair) [19:02:31] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4280985 (10mmodell) ok I'm confused. external_id `6099` doesn't appear to be in the gerrit database, at least not in the account_external_ids table [19:02:47] twentyafterfour your looking in the wrong place! [19:02:49] :) [19:02:56] it's in the All-Users git repo now [19:03:30] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4280987 (10Paladox) @mmodell see https://phabricator.wikimedia.org/T197083#4280194 (we migrated to notedb so it's now in All-Users instead of the db) [19:12:01] paladox: thanks, found it [19:18:03] 10Beta-Cluster-Infrastructure: Puppet failure on deployment-maps03 - https://phabricator.wikimedia.org/T196197#4281034 (10Krenair) 05Open>03Resolved a:03Krenair Set the value to false, it works and has caught up on the last couple weeks worth of puppet changes. [19:18:42] krenair@deployment-tin:/srv/mediawiki-staging$ git diff [19:18:43] diff --git a/wmf-config/event-schemas b/wmf-config/event-schemas [19:18:43] index 4db9d40d2..b50f4e076 160000 [19:18:43] --- a/wmf-config/event-schemas [19:18:43] +++ b/wmf-config/event-schemas [19:18:45] @@ -1 +1 @@ [19:18:47] -Subproject commit 4db9d40d28d61c53cdbca77059d9a2a6e714af89 [19:18:49] +Subproject commit b50f4e0763f4bfa8abf1c4f3afd43cd6067652af [19:18:51] why has it been left like this? [19:21:01] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281041 (10mmodell) ``` remote: Resolving deltas: 100% (2/2) remote: Counting objects: 38496, done remote: Branch refs/meta/external-ids: remote: You are not allowed to perform... [19:25:48] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281044 (10mmodell) ok I added push rights for the external-ids ref and now I got a ton of errors due to non-unique emails [19:26:04] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281045 (10mmodell) cleaning this up will be difficult [19:28:53] RECOVERY - Puppet errors on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:29:51] 10Beta-Cluster-Infrastructure, 10Scap: 'scap update-interwiki-cache' does not exist on deployment-tin - https://phabricator.wikimedia.org/T197166#4281048 (10MarcoAurelio) Aha, that explains. Thanks. [19:34:49] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281063 (10mmodell) upstream bug report: https://bugs.chromium.org/p/gerrit/issues/detail?id=9001&q=external-ids&colspec=ID%20Type%20Stars%20Milestone%20Status%20Priority%20Own... [19:45:07] twentyafterfour your welcome, did you manage to push (just looking at the task now) [19:45:13] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244#4281091 (10Krenair) So I think to replace it properly we need https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/436431/ and https://gerrit.wiki... [19:45:27] paladox: nope, I got about 200 errors [19:45:36] twentyafterfour which error did you get? [19:45:44] i see something about unique emails [19:45:46] paladox: it looks to me like the migration produced a corrupted notedb [19:45:53] hmm [19:46:00] there are a bunch of these: [19:46:35] remote: error: External ID 'gerrit:user' has an invalid email: user.something@gmail.ccom (just some typos) [19:46:43] hmm [19:46:54] oh i see [19:46:55] ccom [19:47:06] twentyafterfour if you can create a task [19:47:10] i can forward it upstream [19:47:27] and error: Email 'something@something.com' is not unique, it's used by the following external IDs: 'gerrit:someuser', 'mailto:something@something.com' [19:47:45] !log powered off deployment-mx T184244 [19:47:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:47:48] T184244: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244 [19:49:04] twentyafterfour https://gerrit-review.googlesource.com/c/gerrit/+/169970 [19:49:08] that looks like a related fix [19:49:11] though not sure [19:51:25] PROBLEM - Host deployment-mx is DOWN: CRITICAL - Host Unreachable (10.68.17.78) [19:52:42] paladox: does the editting interface return in PolyGerrit at some point? 'cause I don't see it in the new UI? [19:52:52] Hauskatze yep [19:52:56] in 2.16 / 3.0 [19:53:00] and it's really nice too! [19:53:01] good [19:53:25] twentyafterfour i've filled https://bugs.chromium.org/p/gerrit/issues/detail?id=9256 [19:53:36] and tagged it as pri -1 [19:53:44] which i think translate into high [19:53:53] i've also cc'ed ekempin [19:56:06] o/ greg-g, so, re the plan for these mcr patches on test for an extra week. Shall we merge them into master today, so I can test on beta all of tommorrow, and then we can try making this branch at some point tommorrow for the evening? [19:59:34] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4281107 (10Framawiki) >>! In T196812#4275713, @Paladox wrote: > @Framawiki wondering can we get the user notice out so we can do this next monday please? :) [[... [20:00:42] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4281114 (10Paladox) @Framawiki oh sorry forgot to tell you that we have postponed it until after the sre offsite (and then a date agreed with releng). [20:09:48] PROBLEM - SSH on deployment-deploy-01 is CRITICAL: Connection refused [20:13:50] addshore: sounds reasonable yeah [20:15:26] 10Release-Engineering-Team, 10ORES, 10Scoring-platform-team, 10Performance: Try to increase ORES deployment parallelism - https://phabricator.wikimedia.org/T197180#4281166 (10awight) [20:16:09] 10Release-Engineering-Team, 10ORES, 10Scoring-platform-team, 10Performance: Try to increase ORES deployment parallelism - https://phabricator.wikimedia.org/T197180#4281178 (10awight) [20:26:04] Hey, I have a question with accessing deployment-tin.eqiad.wmflabs, I try to ssh there and it fails with `Connection closed by UNKNOWN port 65535` [20:26:14] why do I need that? I' [20:26:19] it was renamed [20:26:20] I'm trying to access http://logstash-beta.wmflabs.org/ [20:26:28] oh nvm [20:26:30] Krenair ^^ [20:26:46] ... are you SSHing to deployment-tin or logging into logstash? [20:26:55] paladox, but probably I still need that ssh [20:27:11] because looks like logstash login/pass is in `/root/secrets.txt` [20:27:15] https://www.mediawiki.org/wiki/Beta_Cluster [20:27:18] ah [20:27:22] Logs from the beta cluster are sent to Logstash and can be seen at logstash-beta.wmflabs.org. This site is currently password-protected, for an account look in: ssh deployment-tin.eqiad.wmflabs sudo cat /root/secrets.txt [20:27:32] what's your username? [20:27:38] pmiazga [20:27:54] yeah you don't have access to deployment-prep hosts [20:27:59] lolol [20:28:08] :), do I need to create a phab ticket? [20:28:18] or can I get the login+pass for logstash-beta somehow? [20:28:18] not necessarily [20:28:22] hang on [20:29:19] raynor, see IRC PM [20:33:17] thanks Krenair, case is solved [20:43:50] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:57:21] 10Release-Engineering-Team (Kanban), 10MW-1.31-release: Release MW 1.31 - https://phabricator.wikimedia.org/T191088#4281319 (10demon) 05Open>03Resolved [[https://releases.wikimedia.org/mediawiki/1.31/|Done]] & [[https://lists.wikimedia.org/pipermail/mediawiki-announce/2018-June/000221.html|announced]] [21:00:40] beta-scap-equiad should be running more slowly not each ten minutes... after it ends then another scap starts [21:01:08] well, if l10nupdate wasn't so slow, they wouldn't be a problem [21:01:29] yep [21:01:48] maybe --nol10nupdate each 10 minutes [21:02:00] and just once each hour or so [21:02:24] (03PS1) 10Reedy: Run scap-beta-eqiad every 20 mins because l10n update is slow :( [integration/config] - 10https://gerrit.wikimedia.org/r/440247 [21:02:54] greg-g: ^ Any objections on doing something like that until we're free of hhvm [21:03:08] I dunno if 20 or 30 is really more appropriate [21:03:45] sigh, I suppose, try with 20 so we don't make the delay too long :( [21:04:22] Well, it's currently 10 minutes && job not running [21:04:50] * Reedy looks how long the runs take [21:04:55] MATHS IT UP YO [21:05:32] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ [21:05:36] Hmm, they only take 5-6 minutes [21:07:53] 8/10 some [21:07:56] below [21:13:47] (03CR) 10MarcoAurelio: [C: 031] Run scap-beta-eqiad every 20 mins because l10n update is slow :( [integration/config] - 10https://gerrit.wikimedia.org/r/440247 (owner: 10Reedy) [21:16:06] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/buildTimeTrend [21:17:40] thcipriani: pfft. behave! [21:17:55] * thcipriani helpful [21:18:00] :D [21:18:14] The graph X axis is hard to read [21:18:38] indeed [21:18:54] oh that's helpful [21:30:58] greg-g: awesome, It will be done :) [21:31:21] greg-g: last time we merged "big scary" MCR stuff we got told we should email some mailing lists. Got any ideas as to which ones we should? [21:32:09] Reedy: you could alsoways do something funky like sync all the code every 5 mins, and then do a l10n rebuild every 20 mins? :P [21:32:19] silly localization [21:32:30] heh [21:32:43] I wonder if we can maths and stuff it [21:33:36] 10Release-Engineering-Team, 10Multi-Content-Revisions (MCR-SDC phase 1), 10User-Addshore: Deploy some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4262430 (10Addshore) a:03Addshore [21:33:44] addshore: wikitech-l and ops@ I suppose [21:33:48] Need to look at wmf-beta-autoupdate.py [21:34:04] Where does wmf-beta-autoupdate.py live... [21:34:29] https://github.com/wikimedia/puppet/blob/production/modules/beta/templates/wmf-beta-autoupdate.py.erb [21:35:21] https://github.com/wikimedia/puppet/blob/production/modules/beta/templates/wmf-beta-autoupdate.py.erb [21:35:24] ffs [21:35:44] greg-g: thanks :) [21:36:12] 10Release-Engineering-Team, 10Multi-Content-Revisions (MCR-SDC phase 1), 10User-Addshore: Deploy some MCR related patches on test / group0 for an extended period - https://phabricator.wikimedia.org/T196585#4281359 (10Addshore) The plan is that the patches will be merged into master during EU morning time tom... [21:36:33] twentyafterfour hopefully ekempin will answer the issue tommror [21:36:33] # This is the poor man auto updating script. We should probably split the [21:36:33] # script in different part and have the jobs trigger each other. [21:36:36] Heh, foresight [21:36:54] apparently you can edit it server side but i doin't know how dangerous that is [21:36:58] or how to do it [21:36:58] Hang on [21:37:02] code update doesn't run l10nupdate [21:37:06] scap does [21:37:07] * Reedy slaps himself [21:37:13] (03Abandoned) 10Reedy: Run scap-beta-eqiad every 20 mins because l10n update is slow :( [integration/config] - 10https://gerrit.wikimedia.org/r/440247 (owner: 10Reedy) [21:38:03] So the commit summary is completely wrong [21:38:22] /usr/bin/scap sync "$JOB_NAME (build $BUILD_DISPLAY_NAME)" [21:38:24] That needs changing [21:38:36] But needs some state/tracking [21:38:53] twentyafterfour backporting that commit seems to be more work then it looks heh [21:39:00] it seems alot of fixes were done on master [21:39:30] but i doin't think any fixes to fix it if you get in this state. [21:43:24] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:57:57] 10Release-Engineering-Team, 10ORES, 10Scoring-platform-team (Current): Document: ORES deployment caused some sort of downtime - https://phabricator.wikimedia.org/T197191#4281401 (10awight) [22:14:38] 10Gerrit, 10Release-Engineering-Team: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281423 (10mmodell) p:05Triage>03High [22:15:28] 10Gerrit, 10Release-Engineering-Team: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281436 (10Paladox) filled upstream at https://bugs.chromium.org/p/gerrit/issues/detail?id=9256 [22:17:23] 10Gerrit, 10Release-Engineering-Team: Unable to edit external-ids ref in notedb due to validation - https://phabricator.wikimedia.org/T197192#4281452 (10mmodell) seems related, apparently wontfix: https://bugs.chromium.org/p/gerrit/issues/detail?id=9001 upstreamed by paladox: https://bugs.chromium.org/p/gerrit... [22:20:00] apergos i am thinking 2.16 has this behavour now [22:20:09] i got the notification on https://gerrit-review.googlesource.com/c/gerrit/+/169970 [22:20:30] ok. so not for us just yet [22:21:17] nope [22:30:18] PROBLEM - Free space - all mounts on integration-slave-docker-1006 is CRITICAL: CRITICAL: integration.integration-slave-docker-1006.diskspace.root.byte_percentfree (<22.22%) [22:45:19] RECOVERY - Free space - all mounts on integration-slave-docker-1006 is OK: OK: All targets OK [23:21:33] 10MediaWiki-Codesniffer: Enforce PHP 7 Unicode codepoint escapes "\u{}" to be uppercase and zero-padded to at least four characters - https://phabricator.wikimedia.org/T197196#4281536 (10matmarex) [23:25:03] Undefined index: scope_opener in [23:25:12] in [23:25:20] home/jenkins/workspace/mwext-testextension-hhvm-composer-jessie/src/extensions/WikibaseLexeme/vendor/mediawiki/mediawiki-codesniffer/MediaWiki/Sniffs/ControlStructures/IfElseStructureSniff.php [23:25:39] looks like the sniffer is not feeling well [23:26:16] Quis custodiet ipsos custodes? [23:27:26] 10Gerrit, 10Release-Engineering-Team: Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083#4281552 (10mmodell) [23:27:30] 10Gerrit, 10Phabricator, 10LDAP: Having difficulty logging into Phabricator via LDAP when multiple accounts returned for username (gerrit: Duplicate users: smccandlish) - https://phabricator.wikimedia.org/T138672#4281551 (10mmodell) [23:30:43] 10Continuous-Integration-Infrastructure, 10MediaWiki-Codesniffer, 10Release-Engineering-Team: IfElseStructureSniff produces Undefined index - https://phabricator.wikimedia.org/T197197#4281557 (10Smalyshev) [23:38:24] I want to apologize for deploying my service over the train today, I didn't check the calendar... I've been meaning to move the Services window on Wednesdays, this is a kick to do so! [23:50:58] <3