[00:16:13] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49248 bytes in 0.520 second response time [01:31:54] 10Gerrit, 10Privacy: Gerrit loads font from fonts.googleapis.com and fonts.gstatic.com - https://phabricator.wikimedia.org/T240264 (10Reedy) >>! In T240264#5725695, @Paladox wrote: > Filed upstream https://bugs.chromium.org/p/gerrit/issues/detail?id=12054 Is this not a dupe of {T239122}, or does that bug only... [01:33:04] 10Gerrit, 10Privacy: Gerrit loads font from fonts.googleapis.com and fonts.gstatic.com - https://phabricator.wikimedia.org/T240264 (10Paladox) That bug is for one part, this one is for a plugin. [01:36:05] 10Gerrit, 10Privacy, 10Upstream: Gerrit loads font from fonts.googleapis.com and fonts.gstatic.com - https://phabricator.wikimedia.org/T240264 (10Reedy) [08:23:25] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201908), 10Patch-For-Review: Gerrit -> GitHub replication not up-to-date - https://phabricator.wikimedia.org/T229945 (10Marostegui) 05Resolved→03Open Re-opening this as looks like it is happening again, if you prefer to have a new... [09:18:30] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201912), 10GitHub-Mirrors: Gerrit replication to GitHub is broken - https://phabricator.wikimedia.org/T240322 (10hashar) [09:18:38] thanks to folks' work on the wikidispatch issue and in particular amir 1, my uploads/captions/dispatch script is now running with replag checks. this means that tomorrow I will likely decrease the edit time from 8 seconds between edits to 7 and see how it goes [09:19:04] for today, I'm keeping things the same but watching to see how things go now that replag checking is enabled [09:19:23] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201908), 10Patch-For-Review: Gerrit -> GitHub replication not up-to-date - https://phabricator.wikimedia.org/T229945 (10hashar) 05Open→03Resolved It is an entirely different issue which affects other repositories (if not all of the... [09:19:29] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201912), 10GitHub-Mirrors: Gerrit replication to GitHub is broken - https://phabricator.wikimedia.org/T240322 (10hashar) p:05Triage→03Unbreak! [09:23:04] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (201908), 10Patch-For-Review: Gerrit -> GitHub replication not up-to-date - https://phabricator.wikimedia.org/T229945 (10Marostegui) Thank you! [09:24:22] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201912), 10GitHub-Mirrors: Gerrit replication to GitHub is broken - https://phabricator.wikimedia.org/T240322 (10hashar) From #gerrit replication logs on gerrit1001.eqiad.wmnet in /var/log/gerrit/replication* logs... [09:34:30] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201912), 10GitHub-Mirrors: Gerrit replication to GitHub is broken - https://phabricator.wikimedia.org/T240322 (10hashar) The first error was on 2019-12-09 23:45:43 UTC. From https://github.com/organizations/wikim... [09:48:22] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201912), 10GitHub-Mirrors: Gerrit replication to GitHub is broken - https://phabricator.wikimedia.org/T240322 (10hashar) 05Open→03Resolved a:03hashar Fixed by removing OAuthApp restrictions in Github: T23499... [10:11:25] 10Gerrit: Gerrit replication to Github stalled - https://phabricator.wikimedia.org/T87248 (10hashar) OAuth got enabled again via {T234991} which again broke replication {T240322} :] [10:13:03] (03PS1) 10Ladsgroup: Add Alaa Sarhan to trusted contributors [integration/config] - 10https://gerrit.wikimedia.org/r/556149 [10:15:49] (03CR) 10Sarhan: [C: 03+1] "I confirm wanting to be added and continue contributing :)" [integration/config] - 10https://gerrit.wikimedia.org/r/556149 (owner: 10Ladsgroup) [10:19:10] (03CR) 10MaxSem: "I propose that we merge this. While --debug-tests is indeed still used in old releases we still support, it's an optional flag that doesn'" [integration/quibble] - 10https://gerrit.wikimedia.org/r/547843 (https://phabricator.wikimedia.org/T192167) (owner: 10Jforrester) [10:25:07] Hey anyone is around to merge and deploy this? https://gerrit.wikimedia.org/r/c/integration/config/+/556149 Pretty please? [10:25:50] (03PS2) 10Ladsgroup: Add Alaa Sarhan to trusted contributors [integration/config] - 10https://gerrit.wikimedia.org/r/556149 [10:48:03] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10DC-Ops, 10Operations, 10hardware-requests: Replacement hardware for buster/stretch upgrade of contint1001 and contint2001 - https://phabricator.wikimedia.org/T239880 (10hashar) `contint1001` has 4 1TB SSD: ` # lshw -class disk -short H/W path... [13:11:29] !log integration: rearmed keyholder on integration-cumin [13:11:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:16:46] !log Update Jenkins jobs to drop the m4executor label https://gerrit.wikimedia.org/r/#/c/integration/config/+/556039/ # T225031 [13:16:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:16:49] T225031: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 [13:20:06] (03CR) 10Hashar: [C: 03+2] "Deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/556039 (https://phabricator.wikimedia.org/T225031) (owner: 10Hashar) [13:20:33] !log Removing m4executor labels from Jenkins agents # T225031 [13:20:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:20:58] (03Merged) 10jenkins-bot: Drop m4executor Jenkins label [integration/config] - 10https://gerrit.wikimedia.org/r/556039 (https://phabricator.wikimedia.org/T225031) (owner: 10Hashar) [13:24:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10hashar) `m4executor` is gone! https://integration.wikime... [13:24:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10hashar) [13:33:18] 10Continuous-Integration-Infrastructure, 10Zuul, 10Patch-For-Review: Add support for ecdsa keys in zuul (Also update paramiko to 2.2+) - https://phabricator.wikimedia.org/T171165 (10hashar) [14:12:17] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Epic, 10User-zeljkofilipin: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740 (10hashar) @zeljkofilipin can you update the description of this task?... [14:12:51] 10Phabricator, 10User-Addshore: Phatality phabricator tasks do not have the points field - https://phabricator.wikimedia.org/T240343 (10Addshore) [14:13:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10hashar) [14:14:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10hashar) [14:14:22] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10observability: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I believe... [14:15:38] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (10hashar) [15:17:40] (03CR) 10Kosta Harlan: "> Patch Set 2: Code-Review-1" (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/555750 (owner: 10Kosta Harlan) [15:18:43] (03CR) 10Kosta Harlan: Add php-fpm/apache image for use with MediaWiki Dev Env (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/555914 (https://phabricator.wikimedia.org/T238224) (owner: 10Kosta Harlan) [17:22:49] (03PS1) 10Volans: Setup CI for operations/software/netbox-extras [integration/config] - 10https://gerrit.wikimedia.org/r/556228 (https://phabricator.wikimedia.org/T233183) [17:33:50] happy to say that the replag check seems to be working just fine. tomorrow I'll likely run with a seven second wait between edits, down from 8 [17:34:01] and do that for the rest of the week, see how it goes [18:00:36] (03PS2) 10Dduvall: Avoid extra image layers due to sparse user logs [blubber] - 10https://gerrit.wikimedia.org/r/555573 [18:12:04] (03CR) 10Dduvall: [C: 03+2] Avoid extra image layers due to sparse user logs [blubber] - 10https://gerrit.wikimedia.org/r/555573 (owner: 10Dduvall) [18:12:14] (03CR) 10Krinkle: "Also, we could backport the master patch that made it the default. I don't think we need to worry about unofficial output formatting compa" [integration/quibble] - 10https://gerrit.wikimedia.org/r/547843 (https://phabricator.wikimedia.org/T192167) (owner: 10Jforrester) [18:14:31] (03Merged) 10jenkins-bot: Avoid extra image layers due to sparse user logs [blubber] - 10https://gerrit.wikimedia.org/r/555573 (owner: 10Dduvall) [19:33:09] (03CR) 10Jforrester: [C: 03+2] Add Alaa Sarhan to trusted contributors [integration/config] - 10https://gerrit.wikimedia.org/r/556149 (owner: 10Ladsgroup) [19:33:29] (03CR) 10Jforrester: [C: 03+2] Setup CI for operations/software/netbox-extras [integration/config] - 10https://gerrit.wikimedia.org/r/556228 (https://phabricator.wikimedia.org/T233183) (owner: 10Volans) [19:33:58] (03Merged) 10jenkins-bot: Add Alaa Sarhan to trusted contributors [integration/config] - 10https://gerrit.wikimedia.org/r/556149 (owner: 10Ladsgroup) [19:34:17] (03Merged) 10jenkins-bot: Setup CI for operations/software/netbox-extras [integration/config] - 10https://gerrit.wikimedia.org/r/556228 (https://phabricator.wikimedia.org/T233183) (owner: 10Volans) [19:51:26] (03PS1) 10Dduvall: Fix missing mediawiki/core branch creation [tools/release] - 10https://gerrit.wikimedia.org/r/556242 [20:06:04] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10observability: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520 (10hashar) 05Resolved→03Open I explained it in the task, the... [20:10:33] hashar: i hear you want to talk to me about curl on docs.wikimedia.org? [20:25:48] (03CR) 10Brennen Bearnes: "(Sorry for delayed response here, the wmf.8 train ate my brain for roughly a week.)" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/555750 (owner: 10Kosta Harlan) [20:35:54] (03CR) 10Brennen Bearnes: Add php-fpm/apache image for use with MediaWiki Dev Env (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/555914 (https://phabricator.wikimedia.org/T238224) (owner: 10Kosta Harlan) [20:50:19] Jdlrobson: eek sorry I was in email processing mood :D [20:50:25] and missed the notification [20:56:29] Jdlrobson: but in short from the task, I would guess that either: [20:56:34] A) you get the foreign resources committed in MobileFrontend [20:56:42] > but your change actually get rid of them [20:56:43] or [20:57:08] B) have the Storybook to refers to material on the local machine using relative paths [20:57:39] then a CI job would git clone the dependencies [20:57:57] which for MobileFrontend can be hardcoded as being mediawiki/core and MinervaNeue. That is a good start [20:58:17] we can then look at how those storybooks dependencies can be expressed in the MobileFrontend for CI to discover and process them [20:58:28] but I think that is what I wrote on the task ;] [21:02:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Jenkins: Upgrade Jenkins to 2.190.3 - https://phabricator.wikimedia.org/T239985 (10hashar) [21:07:01] B would be better :) [21:07:19] (03PS1) 10Jforrester: jjb: Drop selenium-MinervaNeue and -QuickSurveys daily ruby jobs [integration/config] - 10https://gerrit.wikimedia.org/r/556252 (https://phabricator.wikimedia.org/T220035) [21:07:19] do you want me to adapt the patch to do that so you have a patch to test on? [21:09:46] (03PS1) 10Jforrester: jjb: Provide selenium-daily-beta-QuickSurveys [integration/config] - 10https://gerrit.wikimedia.org/r/556253 [21:11:28] (03PS1) 10Jayprakash12345: Edit Project Config [labs/tools/bookreader] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/556254 [21:13:44] (03Abandoned) 10Jayprakash12345: Edit Project Config [labs/tools/bookreader] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/556254 (owner: 10Jayprakash12345) [21:23:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201912), 10Jenkins: Upgrade Jenkins to 2.190.3 - https://phabricator.wikimedia.org/T239985 (10hashar) @brennen self proposed to conduct the upgrade with my guidance. I have done so... [21:24:52] (03PS2) 10Jforrester: jjb: Drop selenium-MinervaNeue and -QuickSurveys daily ruby jobs [integration/config] - 10https://gerrit.wikimedia.org/r/556252 (https://phabricator.wikimedia.org/T220035) [21:24:54] (03PS2) 10Jforrester: jjb: Provide selenium-daily-beta-QuickSurveys [integration/config] - 10https://gerrit.wikimedia.org/r/556253 [21:24:56] (03PS1) 10Jforrester: layout: Add Wugapodes to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/556257 [21:25:42] (03CR) 10Jforrester: [C: 03+2] layout: Add Wugapodes to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/556257 (owner: 10Jforrester) [21:26:33] (03Merged) 10jenkins-bot: layout: Add Wugapodes to CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/556257 (owner: 10Jforrester) [21:27:03] !log Zuul: Add Wugapodes to CI whitelist. [21:27:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:30:52] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Investigate gerrit session expiration - https://phabricator.wikimedia.org/T222472 (10thcipriani) [21:32:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Grant "contint-roots" and "releasers-mediawiki" to user brennen - https://phabricator.wikimedia.org/T240382 (10Jdforrester-WMF) [21:34:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Grant "contint-roots" and "releasers-mediawiki" to user brennen - https://phabricator.wikimedia.org/T240382 (10brennen) For purposes of T239985, a Jenkins upgrade. [21:38:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10Operations, 10SRE-Access-Requests: Grant "contint-roots" and "releasers-mediawiki" to user brennen - https://phabricator.wikimedia.org/T240382 (10hashar) [22:09:27] (03CR) 10Jforrester: "```" [tools/release] - 10https://gerrit.wikimedia.org/r/556242 (owner: 10Dduvall)