[00:07:25] PROBLEM - Host deployment-cumin is DOWN: CRITICAL - Host Unreachable (172.16.5.1) [00:10:36] PROBLEM - Host deployment-cache-upload04 is DOWN: CRITICAL - Host Unreachable (172.16.5.14) [01:00:33] 10Release-Engineering-Team, 10Scap: scap should be LCStore-agnostic - https://phabricator.wikimedia.org/T105683 (10Krinkle) [01:00:58] 10Release-Engineering-Team, 10Scap: scap should be LCStore-agnostic - https://phabricator.wikimedia.org/T105683 (10Krinkle) I recall seeing at some point changeset(s) for Scap that did some work toward this end, but can't find them now. [01:16:16] 10Release-Engineering-Team, 10MediaWiki-Core-Testing: Ensure we're testing appropriately and not over-testing across Wikimedia-deployed code - https://phabricator.wikimedia.org/T221434 (10Jdforrester-WMF) [02:07:00] 10Continuous-Integration-Config, 10Jade, 10Patch-For-Review, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rename JADE->Jade in continuous integration - https://phabricator.wikimedia.org/T212181 (10Harej) a:03Ladsgroup [02:07:10] 10Continuous-Integration-Config, 10Jade, 10Patch-For-Review, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rename JADE->Jade in continuous integration - https://phabricator.wikimedia.org/T212181 (10Harej) p:05Normal→03High [03:13:29] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [08:11:59] (03PS4) 10Hashar: Rename mediawiki extension JADE to Jade [integration/config] - 10https://gerrit.wikimedia.org/r/480285 (https://phabricator.wikimedia.org/T212181) (owner: 10Awight) [08:12:36] (03CR) 10Hashar: [C: 03+2] "This CI change was probably just waiting for the repository to be renamed." [integration/config] - 10https://gerrit.wikimedia.org/r/480285 (https://phabricator.wikimedia.org/T212181) (owner: 10Awight) [08:14:45] (03Merged) 10jenkins-bot: Rename mediawiki extension JADE to Jade [integration/config] - 10https://gerrit.wikimedia.org/r/480285 (https://phabricator.wikimedia.org/T212181) (owner: 10Awight) [08:15:31] 10Continuous-Integration-Config, 10Jade, 10Patch-For-Review, 10Scoring-platform-team (Current), 10User-Ladsgroup: Rename JADE->Jade in continuous integration - https://phabricator.wikimedia.org/T212181 (10hashar) 05Open→03Resolved [08:25:42] 10Gerrit: Please rewrite sync-with-gerrit.py to use Gerrit REST API - https://phabricator.wikimedia.org/T194318 (10hashar) p:05Triage→03Low [08:31:33] 10Gerrit: Gerrit: cannot assign username "aldnonymous" to account XXX; name already in use - https://phabricator.wikimedia.org/T221440 (10hashar) [08:32:13] 10Gerrit, 10LDAP: Cannot log into Gerrit: "Cannot assign user name to account; name already in use." - https://phabricator.wikimedia.org/T220867 (10hashar) @Aldnonymous : I have filled that as T221440 [08:32:33] 10Gerrit: Gerrit: cannot assign username "aldnonymous" to account XXX; name already in use - https://phabricator.wikimedia.org/T221440 (10hashar) [08:32:35] 10Gerrit, 10LDAP: Cannot log into Gerrit: "Cannot assign user name to account; name already in use." - https://phabricator.wikimedia.org/T220867 (10hashar) [08:33:58] 10Gerrit, 10LDAP: Gerrit: Cannot assign user name "vladi2016" to account XXXX; name already in use. - https://phabricator.wikimedia.org/T220867 (10hashar) [08:38:33] 10Gerrit, 10LDAP: Gerrit: Cannot assign user name "vladi2016" to account XXXX; name already in use. - https://phabricator.wikimedia.org/T220867 (10hashar) In short: this is a known issue we are encountering with Gerrit due to a migration that has been fully completed. There is a way to fix it on the server but... [09:17:44] (03PS2) 10Hashar: Disable selenium tests for BlueSpicePrivacy [integration/config] - 10https://gerrit.wikimedia.org/r/504246 (owner: 10Robert Vogel) [09:17:47] (03CR) 10Hashar: Disable selenium tests for BlueSpicePrivacy (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/504246 (owner: 10Robert Vogel) [09:18:04] (03CR) 10Hashar: [C: 03+2] Disable selenium tests for BlueSpicePrivacy [integration/config] - 10https://gerrit.wikimedia.org/r/504246 (owner: 10Robert Vogel) [09:20:03] (03CR) 10Hashar: [C: 03+2] [ActiveAbstract] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502842 (owner: 10Umherirrender) [09:20:55] (03Merged) 10jenkins-bot: Disable selenium tests for BlueSpicePrivacy [integration/config] - 10https://gerrit.wikimedia.org/r/504246 (owner: 10Robert Vogel) [09:22:04] (03Merged) 10jenkins-bot: [ActiveAbstract] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502842 (owner: 10Umherirrender) [09:22:29] 10MediaWiki-Codesniffer: RfC: self vs for class self-references - https://phabricator.wikimedia.org/T221236 (10Umherirrender) It seems the sniff `Squiz.Classes.SelfMemberReference` only looks for "self::", not for new or instanceof or phpdocs [09:26:13] (03CR) 10Hashar: [C: 03+2] [ArticlePlaceholder] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502306 (owner: 10Umherirrender) [09:29:59] (03Merged) 10jenkins-bot: [ArticlePlaceholder] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502306 (owner: 10Umherirrender) [09:42:13] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10Services (next): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10Joe) FWIW, I don't think we need the TLS configuration in beta. I can try to simplify things. Sorry for not noticing this bug earlier, but adding #oper... [09:43:00] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10Services (next): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10Joe) a:03Joe [11:01:00] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10MediaWiki-Search, 10Services (next): Beta Cluster search box displays unexisting pages as results - https://phabricator.wikimedia.org/T186993 (10Joe) [11:01:14] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10Patch-For-Review, 10Services (next): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10Joe) 05Open→03Resolved I fixed the configuration of cpjobqueue in deployment-prep, restarted the service, and verified reques... [11:17:26] (03CR) 10Hashar: [C: 03+2] [Calendar] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502319 (owner: 10Umherirrender) [11:17:41] (03CR) 10Hashar: [C: 03+2] [CharInsert] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502322 (owner: 10Umherirrender) [11:17:49] (03CR) 10Hashar: [C: 03+2] [Citoid] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502326 (owner: 10Umherirrender) [11:18:02] (03CR) 10Hashar: [C: 03+2] [CodeMirror] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502572 (owner: 10Umherirrender) [11:18:21] (03CR) 10Hashar: [C: 03+2] [CongressLookup] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502576 (owner: 10Umherirrender) [11:19:20] (03Merged) 10jenkins-bot: [Calendar] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502319 (owner: 10Umherirrender) [11:19:23] (03Merged) 10jenkins-bot: [CharInsert] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502322 (owner: 10Umherirrender) [11:19:26] (03Merged) 10jenkins-bot: [Citoid] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502326 (owner: 10Umherirrender) [11:19:53] (03Merged) 10jenkins-bot: [CodeMirror] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502572 (owner: 10Umherirrender) [11:19:55] (03Merged) 10jenkins-bot: [CongressLookup] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502576 (owner: 10Umherirrender) [11:25:13] (03CR) 10Hashar: [C: 03+2] [Dashiki] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502580 (owner: 10Umherirrender) [11:25:20] (03CR) 10Hashar: [C: 03+2] [DoubleWiki] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502602 (owner: 10Umherirrender) [11:25:25] (03CR) 10Hashar: [C: 03+2] [DynamicSidebar] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502604 (owner: 10Umherirrender) [11:27:37] (03Merged) 10jenkins-bot: [Dashiki] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502580 (owner: 10Umherirrender) [11:27:39] (03Merged) 10jenkins-bot: [DoubleWiki] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502602 (owner: 10Umherirrender) [11:28:02] (03Merged) 10jenkins-bot: [DynamicSidebar] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502604 (owner: 10Umherirrender) [11:40:31] (03CR) 10Hashar: [C: 03+2] [ContactPage] Add ConfirmEdit as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/502577 (owner: 10Umherirrender) [11:41:09] (03CR) 10Hashar: [C: 03+2] [Campaigns] Add dependency to MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/502320 (owner: 10Umherirrender) [11:41:50] (03CR) 10Hashar: [C: 03+2] [WikimediaMessages] Add dependencies for used classes [integration/config] - 10https://gerrit.wikimedia.org/r/500116 (owner: 10Umherirrender) [11:41:58] (03Merged) 10jenkins-bot: [ContactPage] Add ConfirmEdit as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/502577 (owner: 10Umherirrender) [11:42:35] (03Merged) 10jenkins-bot: [Campaigns] Add dependency to MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/502320 (owner: 10Umherirrender) [11:43:40] (03Merged) 10jenkins-bot: [WikimediaMessages] Add dependencies for used classes [integration/config] - 10https://gerrit.wikimedia.org/r/500116 (owner: 10Umherirrender) [12:05:10] (03PS1) 10Umherirrender: [ContributionCredits] Add quibble [integration/config] - 10https://gerrit.wikimedia.org/r/505229 [12:09:22] (03CR) 10Hashar: [C: 03+2] [ContributionCredits] Add quibble [integration/config] - 10https://gerrit.wikimedia.org/r/505229 (owner: 10Umherirrender) [12:11:17] (03Merged) 10jenkins-bot: [ContributionCredits] Add quibble [integration/config] - 10https://gerrit.wikimedia.org/r/505229 (owner: 10Umherirrender) [12:25:37] 10Scap: Scap should only sync built CDB files to production appserver hosts, not the build files as well - https://phabricator.wikimedia.org/T221428 (10thcipriani) 05Open→03Invalid We don't actually sync the cdb files https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/scap/tasks.py#... [13:49:03] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10akosiaris) Before we move forward and enable this, let's make sure we have understood the security repercussions and have mitigated them (an... [14:20:50] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Peachey88) I don't know the gerrit permissions model/system, But could we perhaps only enable it for a small group, so anyone that wants it... [14:21:51] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Paladox) That's not how that works unfortunately, it's either enable it for all, or disable it for all (since this is enabled though a gerri... [15:04:39] guess he must be bt fon :P [15:05:34] he's in the us, so I'm guessing not [15:05:39] fon = fone = phone [15:05:40] probably [15:06:12] heh [16:38:59] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10Patch-For-Review, 10Services (next): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10DannyS712) @Joe I just tried to send a mass message on the beta cluster, and the message was "queued" but isn't actually being de... [16:46:36] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-TranslationNotifications: l10n-update on beta breaks due to: ExtensionDependencyError TranslationNotifications requires Translate to be installed. - https://phabricator.wikimedia.org/T221354 (10sbassett) Is it worth updating https://www.mediawiki.org/wiki/M... [17:55:26] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.34.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T220726 (10mmodell) 05Open→03Resolved [17:57:56] (03PS1) 10Umherirrender: [CrawlableAllPages] Add quibble [integration/config] - 10https://gerrit.wikimedia.org/r/505270 [18:19:33] Q: [18:19:46] if I want to make a change to beta configs in InitializeSettings-labs.php [18:20:10] do I need to scap-sync that file in prod? [18:20:12] after merge? [18:38:35] Ideally, yeah [18:46:19] k will do tuesday then, would be nice to test in beta today but s'ok [18:47:05] ottomata: Deploying IS-labs.php in prod is a noop for prod [18:47:19] yeah you think i'd be ok for me to do now then? [18:47:39] Yeah [18:48:34] ok [18:58:42] +1 [19:27:05] 10Scap: Scap should only sync built CDB files to production appserver hosts, not the build files as well - https://phabricator.wikimedia.org/T221428 (10bd808) Pedantic historian note: the switch from syncing the CDB binary files to syncing the json data files was done all the way back in the [[https://www.mediaw... [19:30:18] Reedy: I have 3 files to sync, didn't realize. still should be no-op [19:30:21] i just merged [19:30:24] but [19:30:25] i want to be safe [19:30:26] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/505280 [19:30:36] ok to sync or should we be safe and not do on friday? [19:30:46] re ebernhardson ^ too [19:32:22] ottomata: In theory, as long as you sync things in the right order... [19:33:30] i think order will likely not matter too much here, but just in case: [19:33:39] Yes it does [19:33:50] InitialiseSettings.php , then -labs.php, then CirrusSearch-common.php [19:33:53] If you sync CirrusSearch-common.php before IS... [19:33:59] You'll get a lot of undefined variable warnings [19:33:59] undefined warnings, ya? [19:34:01] ya [19:34:12] But yeah, as you propose is correct :) [19:34:34] ok proceeding. the code that uses these vars is not live in prod at all, so this should just set some unused globals there [19:41:54] looks fine thank you! [19:51:52] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10thcipriani) >>! In T218750#5125231, @akosiaris wrote: > Before we move forward and enable this, let's make sure we have understood the secur... [19:52:44] fatal: unable to access 'http://tin.eqiad.wmnet/phabricator/deployment/.git/': Could not resolve host: tin.eqiad.wmnet [19:52:53] new host and that issue again, hah [19:53:05] tin. shut up :) [19:53:38] kind of remembers the issue but not fully, but i think we just edited the right file in deploy repo to deploy1001 [19:56:47] !log phab1003 - editing /srv/deployment/phabricator/deployment-cache/.config manually to replace tin.eqiad.wmnet with deploy1001.eqiad.wmnet to fix git cloning issue on first puppet run on new host where somehow tin.eqiad still shows up. fixes puppet run on T221389 [19:56:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:56:50] T221389: setup/install WMF7426 as phab1003.eqiad.wmnet - https://phabricator.wikimedia.org/T221389 [19:57:05] mutante: :) yup [19:57:30] greg-g: fixed! but i dont know how we can remove tin.eqiad forever :) [19:58:54] greg-g: that's a phab replacement server on stretch..that also has enough RAM .. we can finally upgrade phab1001 while having this [19:59:09] follow-up to a meeting we once had [19:59:28] yeah, nice, thanks! [20:01:46] 10Project-Admins: Add task to my personal workboard when I get assigned to the task - https://phabricator.wikimedia.org/T221479 (10D3r1ck01) [20:10:48] 10Phabricator: Add task to my personal workboard when I get assigned to the task - https://phabricator.wikimedia.org/T221479 (10greg) [20:30:03] 10Gerrit, 10Release-Engineering-Team, 10VPS-project-libraryupgrader: Re-enable use of Gerrit HTTP token to push patchsets - https://phabricator.wikimedia.org/T218750 (10Urbanecm) >>! In T218750#5126127, @thcipriani wrote: >>>! In T218750#5125231, @akosiaris wrote: >> This is a double edge knife feature as it... [20:30:44] 10Phabricator: Add task to my personal workboard when I get assigned to the task - https://phabricator.wikimedia.org/T221479 (10D3r1ck01) Thanks for the positioning it right @greg [21:35:47] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 64.29% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:49:57] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:03:19] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10Patch-For-Review, 10Services (next): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10mobrovac) >>! In T215339#5125634, @DannyS712 wrote: > @Joe I just tried to send a mass message on the beta cluster, and the messa... [22:03:52] 10Beta-Cluster-Infrastructure, 10SDC General, 10Wikidata, 10serviceops, 10Services (done): No jobs running on beta cluster - https://phabricator.wikimedia.org/T215339 (10mobrovac) [22:19:07] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:27:53] ya'll are busy today ^ [22:35:24] 10Release-Engineering-Team, 10Scap: scap should be LCStore-agnostic - https://phabricator.wikimedia.org/T105683 (10bd808) >>! In T105683#5124588, @Krinkle wrote: > I recall seeing at some point changeset(s) for Scap that did some work toward this end, but can't find them now. It was (sort of) implemented in h... [22:42:12] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:00:50] (03PS2) 10Dduvall: experimental: Support buildkit [blubber] - 10https://gerrit.wikimedia.org/r/504651 [23:02:24] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [blubber] - 10https://gerrit.wikimedia.org/r/504651 (owner: 10Dduvall) [23:02:26] (03CR) 10jerkins-bot: [V: 04-1] experimental: Support buildkit [blubber] - 10https://gerrit.wikimedia.org/r/504651 (owner: 10Dduvall) [23:05:39] (03PS3) 10Dduvall: experimental: Support buildkit [blubber] - 10https://gerrit.wikimedia.org/r/504651 [23:07:18] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [blubber] - 10https://gerrit.wikimedia.org/r/504651 (owner: 10Dduvall) [23:07:20] (03CR) 10jerkins-bot: [V: 04-1] experimental: Support buildkit [blubber] - 10https://gerrit.wikimedia.org/r/504651 (owner: 10Dduvall)