[00:20:17] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [00:20:17] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [00:30:16] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [00:30:16] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [01:12:20] 10Release-Engineering-Team, 10MediaWiki-API, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson)... [01:12:20] 10Release-Engineering-Team, 10MediaWiki-API, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson)... [01:46:35] 10Release-Engineering-Team, 10MediaWiki-API, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (... [01:46:35] 10Release-Engineering-Team, 10MediaWiki-API, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (... [01:54:11] 10Release-Engineering-Team, 10MediaWiki-API, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (... [01:54:11] 10Release-Engineering-Team, 10MediaWiki-API, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (... [01:57:26] 10Release-Engineering-Team, 10MediaWiki-API, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (... [01:57:26] 10Release-Engineering-Team, 10MediaWiki-API, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (... [05:42:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 (10greg) p:05Triage>03Normal [05:42:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 (10greg) p:05Triage>03Normal [06:17:14] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:17:14] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:24:36] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Cleanup, 10Repository-Admins, 10Scoring-platform-team: Investigate and clean up deprecated ORES repos - https://phabricator.wikimedia.org/T179610 (10Ladsgroup) p:05Triage>03Low [06:24:36] 10Gerrit, 10Release-Engineering-Team (Watching / External), 10Cleanup, 10Repository-Admins, 10Scoring-platform-team: Investigate and clean up deprecated ORES repos - https://phabricator.wikimedia.org/T179610 (10Ladsgroup) p:05Triage>03Low [06:27:16] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [06:27:16] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [08:15:28] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet fail on deployment-mediawiki-07, missing private hiera variable - https://phabricator.wikimedia.org/T210497 (10Joe) So to explain what happened here: - `labs/private` works a double function, as a dupe for the production puppet repo for the compiler (which uses... [08:15:28] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet fail on deployment-mediawiki-07, missing private hiera variable - https://phabricator.wikimedia.org/T210497 (10Joe) So to explain what happened here: - `labs/private` works a double function, as a dupe for the production puppet repo for the compiler (which uses... [08:18:01] (03CR) 10Hashar: [C: 032] Add job for urbanecmbot [integration/config] - 10https://gerrit.wikimedia.org/r/475905 (owner: 10Urbanecm) [08:18:01] (03CR) 10Hashar: [C: 032] Add job for urbanecmbot [integration/config] - 10https://gerrit.wikimedia.org/r/475905 (owner: 10Urbanecm) [08:18:59] !log test mediawiki kafka logging on deployment-mediawiki-07 [08:18:59] !log test mediawiki kafka logging on deployment-mediawiki-07 [08:19:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:19:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:20:39] (03Merged) 10jenkins-bot: Add job for urbanecmbot [integration/config] - 10https://gerrit.wikimedia.org/r/475905 (owner: 10Urbanecm) [08:20:39] (03Merged) 10jenkins-bot: Add job for urbanecmbot [integration/config] - 10https://gerrit.wikimedia.org/r/475905 (owner: 10Urbanecm) [08:30:50] (03PS1) 10Hashar: Migrate Castor to integration-castor03 [integration/config] - 10https://gerrit.wikimedia.org/r/476213 (https://phabricator.wikimedia.org/T208803) [08:30:50] (03PS1) 10Hashar: Migrate Castor to integration-castor03 [integration/config] - 10https://gerrit.wikimedia.org/r/476213 (https://phabricator.wikimedia.org/T208803) [08:32:47] (03CR) 10Hashar: [C: 032] Migrate Castor to integration-castor03 [integration/config] - 10https://gerrit.wikimedia.org/r/476213 (https://phabricator.wikimedia.org/T208803) (owner: 10Hashar) [08:32:47] (03CR) 10Hashar: [C: 032] Migrate Castor to integration-castor03 [integration/config] - 10https://gerrit.wikimedia.org/r/476213 (https://phabricator.wikimedia.org/T208803) (owner: 10Hashar) [08:35:16] (03Merged) 10jenkins-bot: Migrate Castor to integration-castor03 [integration/config] - 10https://gerrit.wikimedia.org/r/476213 (https://phabricator.wikimedia.org/T208803) (owner: 10Hashar) [08:35:17] (03Merged) 10jenkins-bot: Migrate Castor to integration-castor03 [integration/config] - 10https://gerrit.wikimedia.org/r/476213 (https://phabricator.wikimedia.org/T208803) (owner: 10Hashar) [08:45:38] !log Switching Jenkins job cache to integration-castor03 with AN EMPTY CACHE | T208803 [08:45:38] !log Switching Jenkins job cache to integration-castor03 with AN EMPTY CACHE | T208803 [08:45:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:45:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:45:41] T208803: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 [08:45:41] T208803: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 [09:08:02] (03PS2) 10Hashar: Create selenium-daily-beta-AdvancedSearch Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:02] (03PS2) 10Hashar: Create selenium-daily-beta-AdvancedSearch Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460516 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:04] (03PS2) 10Hashar: Create selenium-daily-beta-ElectronPdfService Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460552 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:04] (03PS2) 10Hashar: Create selenium-daily-beta-ElectronPdfService Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460552 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:06] (03PS2) 10Hashar: Create selenium-daily-beta-Newsletter Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:06] (03PS2) 10Hashar: Create selenium-daily-beta-Newsletter Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:08] (03PS2) 10Hashar: Create selenium-daily-beta-ORES Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:08] (03PS2) 10Hashar: Create selenium-daily-beta-ORES Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:10] (03PS2) 10Hashar: Create selenium-daily-beta-TwoColConflict Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:10] (03PS2) 10Hashar: Create selenium-daily-beta-TwoColConflict Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:12] (03PS2) 10Hashar: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:12] (03PS2) 10Hashar: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [09:08:20] zeljkof: rebased as a long chain [09:08:20] zeljkof: rebased as a long chain [09:08:33] zeljkof: we should just merge them all since the job got deployed and do notify people [09:08:33] zeljkof: we should just merge them all since the job got deployed and do notify people [09:09:26] I think I didn't deploy since I was waiting for people to say they want notification [09:09:26] I think I didn't deploy since I was waiting for people to say they want notification [09:15:26] https://integration.wikimedia.org/ci/job/selenium-daily-beta-TwoColConflict/jobConfigHistory/ :) [09:15:26] https://integration.wikimedia.org/ci/job/selenium-daily-beta-TwoColConflict/jobConfigHistory/ :) [09:15:50] you did apparently :) I have cherry picked all of them in a long chain and refreshed all those jobs for a different commit/patch [09:15:50] you did apparently :) I have cherry picked all of them in a long chain and refreshed all those jobs for a different commit/patch [09:17:47] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) CI jobs have been migrated to use `integration-castor03`. I am keeping `castor02` around just in case. [09:17:47] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10hashar) CI jobs have been migrated to use `integration-castor03`. I am keeping `castor02` around just in case. [10:00:24] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10hashar) [10:01:41] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:32] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48507 bytes in 0.828 second response time [10:37:49] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), 10Patch-For-Review: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 (10hashar) > ResourceLoaderFileModule::readStyleFile: style file not found: "/workspace/src/extensions/Vi... [10:40:58] hi! who should I be adding to https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/476228 for signoff ? [12:15:13] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [12:25:18] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [14:29:55] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [14:32:45] 10Release-Engineering-Team (Kanban), 10Code-Health-Metrics, 10Patch-For-Review, 10User-zeljkofilipin: Code health metrics spike - https://phabricator.wikimedia.org/T207046 (10kostajh) @hashar to move this task forward, we need the sonar-scanner image built ([gerrit/475496](https://gerrit.wikimedia.org/r/c/... [14:32:50] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10hashar) [14:37:11] 10Release-Engineering-Team, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError... - https://phabricator.wikimedia.org/T210557 [14:43:13] I'd like to merge https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/476228, anything special I should do? [14:43:22] like engaging folks [14:44:35] hashar: can integration-publishing.integration.eqiad.wmflabs be deleted now, or soon? [14:45:38] andrewbogott: I havent verified the service is working properly yet. Will delete it once checked :) [14:45:52] ok! thank you [14:45:59] andrewbogott: also this morning I have migrated a service that relied on castor02 to a new instance integration-castor03 but that rquires verificatoin as well [14:46:12] I am confident both works, but want to triple check. Once done I will delete them [14:46:27] rest of slaves I thin,k we can just migrate them [14:46:53] I am going out for some duties this afternoon though. But we can sync together tomorrow and migrate the rest? [14:47:06] hopefully tomorrow I will have verified and deleted the publishing and castor instances :) [15:00:26] maybe third time will be a charm: thoughts on how to deploy and/or signoff on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/476228 ? I just do it? [15:17:02] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikimediaIncubator: Create a Wikimedia Incubator test wiki on the Beta Cluster - https://phabricator.wikimedia.org/T210492 (10Ebe123) https://incubator.wmflabs.org/wiki/Main_Page ? [15:35:53] Hey folks [15:36:04] bad news :( I fear we need to roll back the train for Wikidata [15:36:13] we have two really bad issues [15:36:22] tickets are being filed atm [15:38:38] * Lydia_WMDE pokes thcipriani [15:39:33] Lydia_WMDE: just for wikidatawiki? leave it on testwikidatawiki? [15:39:53] thcipriani: don't care too much about testwikidatawiki for those two issues [15:40:05] wikidata itself is where it hurts ;-) [15:42:06] first ticket: https://phabricator.wikimedia.org/T210617 [15:42:08] second coming [15:51:55] https://phabricator.wikimedia.org/T210618 [15:54:36] thanks both! [15:54:54] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: 5 of the 15 prioritized repositories have at least 1 end-to-end test - https://phabricator.wikimedia.org/T206621 (10zeljkofilipin) [15:59:25] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10thcipriani) At 15:49UTC I rolled back wikidatawiki to 1.33.0-wmf.4 at @Lydia_Pintscher's request via IRC. All other group1 wikis are stil... [16:12:13] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10Lucas_Werkmeister_WMDE) [16:15:54] I'm also wondering how evil https://phabricator.wikimedia.org/T210621 is [16:26:30] thcipriani: If you could, we need to revert a patch that was deployed for Incubator too [16:26:42] The reverted patch: https://gerrit.wikimedia.org/r/476278/ [16:27:22] Relevant task: https://phabricator.wikimedia.org/T117023 [16:28:52] Hydriz: sure if you can get that merged to master, I can backport it for wmf.6 for you [16:29:12] I can merge it, but it's my own patch so I don't think I can [16:29:22] Unless this is an exception to the +2? [16:30:42] Oh oops my bad, misinterpreted the policy, merging it now [16:31:27] I think it's more of a "shouldn't" policy rather than a "mustn't" policy :) [16:34:33] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10greg) [16:34:52] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10greg) readding the wikidata blocking task as it is blocking the full rollout of wmf.6 to all wikis (wikidatawiki). [16:38:24] Hydriz: hang on, I was just reading the task, you're seeing a problem on wikipedias? [16:38:49] for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaIncubator/+/347010 ? [16:39:03] thcipriani: No, we mean for Wikipedia info pages on the Incubator wiki [16:39:13] oh! ok. [16:39:55] (now you see why I was hesitant to review your patch :P) [16:40:02] * thcipriani merged backport [16:40:32] thcipriani: Thanks a lot, apologies for the trouble :/ [16:40:48] backport, FYI: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaIncubator/+/476304/ [16:42:02] Hydriz: it happens, and yw :) Once that merges I'll pull it over to mwdebug1002 and have you check to make sure it's working ok there. I'll also probably switch to the #wikimedia-operations channel [16:42:57] Hydriz: https://wikitech.wikimedia.org/wiki/X-Wikimedia-Debug#Browser_extensions in case you're not familiar with the debug servers [16:43:05] Alright sure, thanks! [16:47:19] (03PS1) 10Hydriz: Updating email address [integration/config] - 10https://gerrit.wikimedia.org/r/476305 [16:54:44] Gerrit will need a restart to reload config . .will be right baclk [17:14:52] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10Lucas_Werkmeister_WMDE) [17:30:21] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10Lucas_Werkmeister_WMDE) [17:37:20] 10Deployments, 10Release-Engineering-Team (Kanban), 10User-greg: Outline options for managing incident reports creation, follow-ups, and analysis - https://phabricator.wikimedia.org/T208632 (10zeljkofilipin) The same tool (or a similar one) could be used to create incident reports for train deployments. Each... [17:58:45] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11): JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [18:37:16] (03CR) 10Hashar: [C: 032] Updating email address [integration/config] - 10https://gerrit.wikimedia.org/r/476305 (owner: 10Hydriz) [18:37:16] (03CR) 10Hashar: [C: 032] Updating email address [integration/config] - 10https://gerrit.wikimedia.org/r/476305 (owner: 10Hydriz) [18:39:11] (03Merged) 10jenkins-bot: Updating email address [integration/config] - 10https://gerrit.wikimedia.org/r/476305 (owner: 10Hydriz) [18:39:11] (03Merged) 10jenkins-bot: Updating email address [integration/config] - 10https://gerrit.wikimedia.org/r/476305 (owner: 10Hydriz) [18:43:23] 10Gerrit, 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10hashar) https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475226/ enables logging for plugins: [2018-11-28 16:57:47,182] [main] INFO : Loaded plugin rev... [18:43:23] 10Gerrit, 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10hashar) https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475226/ enables logging for plugins: [2018-11-28 16:57:47,182] [main] INFO : Loaded plugin rev... [19:06:28] 10Release-Engineering-Team, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError... - https://phabricator.wikimedia.org/T210557 [19:06:28] 10Release-Engineering-Team, 10Quibble, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2), 10User-zeljkofilipin: Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError... - https://phabricator.wikimedia.org/T210557 [19:09:52] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10Ladsgroup) [19:09:53] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T206660 (10Ladsgroup) [19:32:04] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [19:32:05] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [19:41:51] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Security: Install docker on releases-jenkins - https://phabricator.wikimedia.org/T208529 (10thcipriani) a:03thcipriani [19:41:51] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Security: Install docker on releases-jenkins - https://phabricator.wikimedia.org/T208529 (10thcipriani) a:03thcipriani [19:54:09] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10Anomie) Apparently the test added for {T209232} is sometimes running out of allowed memor... [19:54:09] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10Anomie) Apparently the test added for {T209232} is sometimes running out of allowed memor... [20:04:44] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [20:04:44] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [20:06:16] (03PS3) 10Hashar: Create selenium-daily-beta-ORES Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:06:16] (03PS3) 10Hashar: Create selenium-daily-beta-ORES Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:06:18] (03PS3) 10Hashar: Create selenium-daily-beta-TwoColConflict Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:06:18] (03PS3) 10Hashar: Create selenium-daily-beta-TwoColConflict Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:06:20] (03PS3) 10Hashar: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:06:20] (03PS3) 10Hashar: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:07:15] (03CR) 10Hashar: Create selenium-daily-beta-ORES Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:07:15] (03CR) 10Hashar: Create selenium-daily-beta-ORES Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [20:13:03] 10Beta-Cluster-Infrastructure, 10Operations, 10media-storage, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236 (10Dzahn) rebased, amended. is it like the cherry-pick though? https://gerrit.wikimedia.org... [20:13:03] 10Beta-Cluster-Infrastructure, 10Operations, 10media-storage, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236 (10Dzahn) rebased, amended. is it like the cherry-pick though? https://gerrit.wikimedia.org... [20:58:04] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10bd808) [20:58:04] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Fix tests around the toollabs repo and packaging - https://phabricator.wikimedia.org/T210663 (10bd808) [21:08:41] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10Bawolff) The gci task is already accepted - we cant force the student to do more work... [21:08:41] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10Bawolff) The gci task is already accepted - we cant force the student to do more work... [21:24:07] (03CR) 1001tonythomas: [C: 031] "Sorry for being late on this. Lets do this." [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [21:24:07] (03CR) 1001tonythomas: [C: 031] "Sorry for being late on this. Lets do this." [integration/config] - 10https://gerrit.wikimedia.org/r/460555 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [21:27:54] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [21:27:54] 10Continuous-Integration-Config, 10JavaScript, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [22:22:32] 10Release-Engineering-Team (Backlog), 10Operations: Point keyholder github mirror to gerrit - https://phabricator.wikimedia.org/T210674 (10thcipriani) p:05Triage>03Normal [22:22:32] 10Release-Engineering-Team (Backlog), 10Operations: Point keyholder github mirror to gerrit - https://phabricator.wikimedia.org/T210674 (10thcipriani) p:05Triage>03Normal [22:25:20] 10Release-Engineering-Team (Kanban), 10Operations: Point keyholder github mirror to gerrit - https://phabricator.wikimedia.org/T210674 (10thcipriani) a:03thcipriani [22:25:20] 10Release-Engineering-Team (Kanban), 10Operations: Point keyholder github mirror to gerrit - https://phabricator.wikimedia.org/T210674 (10thcipriani) a:03thcipriani [22:48:30] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10Anomie) Or you could have someone else try it again. [22:48:30] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata: Scribunto test “LuaStandalone: SecurityTests[1]: CVE-2014-5461” failing in Wikibase CI builds - https://phabricator.wikimedia.org/T210634 (10Anomie) Or you could have someone else try it again. [23:01:21] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476419 [23:01:21] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476419 [23:01:23] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476419 (owner: 10QChris) [23:01:23] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476419 (owner: 10QChris) [23:01:49] (03PS1) 10QChris: Import done. Revoke import grants [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476420 [23:01:49] (03PS1) 10QChris: Import done. Revoke import grants [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476420 [23:01:50] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476420 (owner: 10QChris) [23:01:50] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [research/article-recommender] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/476420 (owner: 10QChris) [23:35:47] is anyone around to debug slowness in the gate-and-submit pipeline? The top of the queue at https://integration.wikimedia.org/zuul/ says it has been running for 1hr 16min. [23:35:47] is anyone around to debug slowness in the gate-and-submit pipeline? The top of the queue at https://integration.wikimedia.org/zuul/ says it has been running for 1hr 16min. [23:38:36] Amir and James discussed it in #mediawiki earlier [23:38:36] Amir and James discussed it in #mediawiki earlier [23:38:39] (I think) [23:38:39] (I think) [23:53:32] from my cursory glance, it looks like we're just overloaded with jobs [23:53:32] from my cursory glance, it looks like we're just overloaded with jobs