[01:14:46] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10Wikimedia-Incident: Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10Aklapper) @Yann: No, because the "Priority" field is not for users to express how... [03:18:00] 10Continuous-Integration-Infrastructure, 10Tracking: doc.wikimedia.org: Generate documentation for release tags (tracking) - https://phabricator.wikimedia.org/T73062 (10Krinkle) [03:18:02] 10Continuous-Integration-Infrastructure, 10Zuul, 10Upstream: Allow ref-updated listener to filter out tag deletions - https://phabricator.wikimedia.org/T96390 (10Krinkle) 05Open→03Declined We only have 1 job that does this currently (train.yaml) and it seems easy enough to copy around to the few jobs tha... [03:18:34] 10Continuous-Integration-Config, 10Librarization, 10MediaWiki-Vendor: Jenkins: Create job for verifying committed "vendor" directory from composer - https://phabricator.wikimedia.org/T74952 (10Krinkle) [03:25:47] 10Continuous-Integration-Infrastructure: Zuul: Highlight relevant change on Zuul status page when following submit pipeline url - https://phabricator.wikimedia.org/T65399 (10Krinkle) >>! In T65399#677079, @gerritbot wrote: > Change 123145 merged by jenkins-bot: > Zuul status page: Add support for highlighting a... [05:02:14] (03Abandoned) 10Giuseppe Lavagetto: Edit Project Config [docker-images/production-images] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/500386 (owner: 10Giuseppe Lavagetto) [05:02:19] (03Abandoned) 10Giuseppe Lavagetto: Edit Project Config [docker-images] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/500385 (owner: 10Giuseppe Lavagetto) [05:45:47] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [05:56:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10Marostegui) [06:55:45] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:01:32] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10Wikimedia-Incident: Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10jcrespo) @Yann and @Aklapper please stop discussing that (or at least discussing... [08:03:06] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10Wikimedia-Incident: Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10jcrespo) [09:39:50] (03PS2) 10Hashar: jjb: Updated Fresnel to 0.2.2 [integration/config] - 10https://gerrit.wikimedia.org/r/500511 (owner: 10Phedenskog) [09:40:38] (03CR) 10Hashar: [C: 03+1] "Rebased and updated to point at releng/quibble-fresnel:0.0.31-2 which has Quibble at the proper version (0.0.31)" [integration/config] - 10https://gerrit.wikimedia.org/r/500511 (owner: 10Phedenskog) [09:44:48] (03PS1) 10Hashar: Switch Jenkins obs to Quibble 0.0.31 [integration/config] - 10https://gerrit.wikimedia.org/r/500682 (https://phabricator.wikimedia.org/T219647) [09:46:00] (03CR) 10Hashar: [C: 03+2] Switch Jenkins obs to Quibble 0.0.31 [integration/config] - 10https://gerrit.wikimedia.org/r/500682 (https://phabricator.wikimedia.org/T219647) (owner: 10Hashar) [09:46:26] !log Upgrading CI Quibble jobs to 0.0.31 [09:46:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:48:53] (03Merged) 10jenkins-bot: Switch Jenkins obs to Quibble 0.0.31 [integration/config] - 10https://gerrit.wikimedia.org/r/500682 (https://phabricator.wikimedia.org/T219647) (owner: 10Hashar) [09:52:43] 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble should errors out when a git submodule fails - https://phabricator.wikimedia.org/T198980 (10hashar) [09:52:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Upgrade CI jobs to Quibble 0.0.31 - https://phabricator.wikimedia.org/T219647 (10hashar) 05Open→03Resolved Should be good now. [09:53:58] 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Quibble should errors out when a git submodule fails - https://phabricator.wikimedia.org/T198980 (10hashar) 05Open→03Resolved Should be good since Qiubble 0.0.31 [09:55:02] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Quibble (marble): Quibble should clone repositories in parallel - https://phabricator.wikimedia.org/T211701 (10hashar) So this is possible since Quibble 0.0.31 (0.0.30 is broken). Still have to update the Jenkins jobs to pass `--git-parallel 8`. [10:20:09] (03PS1) 10Hashar: Clone repos in parallel in wmf-quibble [integration/config] - 10https://gerrit.wikimedia.org/r/500689 (https://phabricator.wikimedia.org/T211701) [10:23:29] (03CR) 10Hashar: [C: 03+2] Clone repos in parallel in wmf-quibble [integration/config] - 10https://gerrit.wikimedia.org/r/500689 (https://phabricator.wikimedia.org/T211701) (owner: 10Hashar) [10:26:29] (03Merged) 10jenkins-bot: Clone repos in parallel in wmf-quibble [integration/config] - 10https://gerrit.wikimedia.org/r/500689 (https://phabricator.wikimedia.org/T211701) (owner: 10Hashar) [10:28:21] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Quibble (marble): Quibble should clone repositories in parallel - https://phabricator.wikimedia.org/T211701 (10hashar) I have updated the wmf-quibble jobs for a starter. Will see later at generalizing the parameter to all jobs and probably just make... [10:47:23] (03CR) 10Alexandros Kosiaris: "Per https://phabricator.wikimedia.org/T205911#5070535, an alternate path was chosen" [blubber] - 10https://gerrit.wikimedia.org/r/492922 (https://phabricator.wikimedia.org/T205911) (owner: 10Alexandros Kosiaris) [10:47:35] (03Abandoned) 10Alexandros Kosiaris: Install heapdump and gc-stats when env production [blubber] - 10https://gerrit.wikimedia.org/r/492922 (https://phabricator.wikimedia.org/T205911) (owner: 10Alexandros Kosiaris) [11:03:21] 10Continuous-Integration-Infrastructure, 10Tracking: doc.wikimedia.org: Generate documentation for release tags (tracking) - https://phabricator.wikimedia.org/T73062 (10hashar) [11:03:24] 10Continuous-Integration-Infrastructure, 10Zuul, 10Upstream: Allow ref-updated listener to filter out tag deletions - https://phabricator.wikimedia.org/T96390 (10hashar) 05Declined→03Open Cool we have a task for this! I am reopening it since @thcipriani looked into this recently for https://gerrit.wikime... [11:03:59] (03PS3) 10Hashar: post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [11:04:09] (03CR) 10jerkins-bot: [V: 04-1] post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [11:16:59] (03CR) 10Hashar: "Thank you for the follow up I have missed the email notification unfortunately. Some nitpicks inline, nothing to worry about though." (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [11:17:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Zuul, 10Patch-For-Review, 10Upstream: Allow ref-updated listener to filter out tag deletions - https://phabricator.wikimedia.org/T96390 (10hashar) a:03thcipriani [11:30:23] _joe_: thank you for the docker-pkg follow up :] [11:31:40] 10Continuous-Integration-Infrastructure, 10docker-pkg, 10Patch-For-Review: docker-pkg is unhappy on contint1001 - https://phabricator.wikimedia.org/T219778 (10hashar) 05Open→03Resolved a:03Joe Tentatively resolved. Thank you very much @Joe [11:32:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Release Pipeline: contint1001:/var/lib/docker growth - https://phabricator.wikimedia.org/T207702 (10hashar) [11:32:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10hashar) [11:41:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10hashar) 05duplicate→03Open p:05Triage→03Unbreak! a:03hashar That task is valid it is for... [12:07:29] (03PS1) 10Hashar: Compress MediaWiki debug logs [integration/config] - 10https://gerrit.wikimedia.org/r/500714 (https://phabricator.wikimedia.org/T219850) [12:08:12] (03CR) 10Hashar: [C: 03+2] Compress MediaWiki debug logs [integration/config] - 10https://gerrit.wikimedia.org/r/500714 (https://phabricator.wikimedia.org/T219850) (owner: 10Hashar) [12:11:02] (03Merged) 10jenkins-bot: Compress MediaWiki debug logs [integration/config] - 10https://gerrit.wikimedia.org/r/500714 (https://phabricator.wikimedia.org/T219850) (owner: 10Hashar) [12:44:55] (03PS1) 10BBlack: operations-dnslint: update to 0.0.5 [integration/config] - 10https://gerrit.wikimedia.org/r/500719 [12:49:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10hashar) 05Open→03Resolved The jobs running MediaWiki tests no gzip the hu... [13:01:45] (03PS2) 10BBlack: operations-dnslint: update to 0.0.5 [integration/config] - 10https://gerrit.wikimedia.org/r/500719 [13:05:56] (03CR) 10Giuseppe Lavagetto: [C: 03+1] operations-dnslint: update to 0.0.5 [integration/config] - 10https://gerrit.wikimedia.org/r/500719 (owner: 10BBlack) [13:09:09] (03CR) 10Volans: [C: 03+1] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/500719 (owner: 10BBlack) [15:08:24] (03CR) 10Jforrester: [C: 03+1] Whitelist user Nikitrain on CI [integration/config] - 10https://gerrit.wikimedia.org/r/500360 (owner: 10MarcoAurelio) [15:58:37] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Testing, 10JavaScript: Make QUnit tests run without installing MediaWiki - https://phabricator.wikimedia.org/T89433 (10Krinkle) [16:07:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): - https://phabricator.wikimedia.org/T219850 (10hashar) With compression of mediawiki debug logs, disk usage went down to 287... [17:00:49] <_joe_> hi people [17:01:13] <_joe_> I'd need to find a charitable soul that wants to merge https://gerrit.wikimedia.org/r/c/integration/config/+/500382 and go do the deed of rebuilding all those images [17:07:45] _joe_: I'm guessing that'd be hashar, he's been working on that issue. Is it ok to wait until tomorrow your time? [17:09:06] <_joe_> greg-g: sure, I'll poke hashar again :) [17:09:18] :) [17:19:44] 10Continuous-Integration-Config, 10Code-Health-Metrics, 10Discovery-Search: Run sonar analysis as a pre-merge step for search platform maven projects - https://phabricator.wikimedia.org/T215135 (10Gehel) [17:46:19] 10Beta-Cluster-Infrastructure, 10Analytics, 10EventBus, 10Patch-For-Review, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10dduvall) p:05Triage→03Unbreak! Marking this UBN as is policy for all deployment block... [17:46:31] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T206678 (10Jdforrester-WMF) [17:50:20] 10Phabricator, 10Project-Admins: Move #Community_Videos_from_the_executive_director under #Community-Relations - https://phabricator.wikimedia.org/T219610 (10Aklapper) [17:50:43] 10Beta-Cluster-Infrastructure, 10Analytics, 10EventBus, 10Patch-For-Review, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10Pchelolo) The https://gerrit.wikimedia.org/r/500363 fixes it. Don't want to self-merge my... [18:09:44] Project beta-scap-eqiad build #243746: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/243746/ [18:10:17] 10Beta-Cluster-Infrastructure, 10Analytics, 10EventBus, 10Patch-For-Review, 10Wikimedia-production-error: PHP Warning: Array key should be either a string or an integer - https://phabricator.wikimedia.org/T219738 (10dduvall) a:03Pchelolo >>! In T219738#5078550, @Pchelolo wrote: > The https://gerrit.wik... [18:20:42] Yippee, build fixed! [18:20:43] Project beta-scap-eqiad build #243747: 09FIXED in 9 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/243747/ [18:50:50] (03PS8) 10Dduvall: WIP Allow configuration of pipeline [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 [18:51:34] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 (owner: 10Dduvall) [18:53:35] (03PS1) 10Santhosh: ContentTranslation: Remove Wikibase and GuidedTour CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) [19:00:11] (03CR) 10Jforrester: [C: 03+1] ContentTranslation: Remove Wikibase and GuidedTour CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [19:03:20] PROBLEM - SSH on integration-slave-docker-1053 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:00] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10Wikimedia-Incident: Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10greg) This is a great example of a almost-worst case scenario, sadly. Things tha... [19:07:05] (03PS4) 10Ejegg: DonationInterface tests run on PHP7 and MediaWiki 1.31 [integration/config] - 10https://gerrit.wikimedia.org/r/499334 [19:08:23] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T206678 (10Jdforrester-WMF) [19:10:14] (03CR) 10KartikMistry: [C: 03+1] ContentTranslation: Remove Wikibase and GuidedTour CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [19:11:04] 10Continuous-Integration-Config, 10MediaWiki-extensions-Scribunto, 10Wikidata, 10Patch-For-Review, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): [Task] Add Scribunto to extension-gate in CI - https://phabricator.wikimedia.org/T125050 (10greg) >>! In T125050#5074872, @WMDE-leszek wrote: > So this t... [19:23:07] (03CR) 10Krinkle: [C: 04-1] DonationInterface tests run on PHP7 and MediaWiki 1.31 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/499334 (owner: 10Ejegg) [19:24:49] (03PS5) 10Ejegg: DonationInterface tests run on PHP7 and MediaWiki 1.31 [integration/config] - 10https://gerrit.wikimedia.org/r/499334 [19:25:09] (03CR) 10Ejegg: "Thanks for the review, Krinkle. Fixed" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/499334 (owner: 10Ejegg) [19:28:19] RECOVERY - SSH on integration-slave-docker-1053 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u8 (protocol 2.0) [19:37:37] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Browser-Tests, 10Patch-For-Review: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10Jdlrobson) p:05Triage→03Unbreak! Seems to be blocking over extensions/skins e.g. Vector.. argggg [19:47:05] have there been any recent changes to the login workflow that might have broken CI? [19:47:20] we're seeing lots of Jenkins failures on mobile's browser tests as the web driver can no longer login [19:47:35] looking for clues - seems unrelated to MobileFrontend so likely a change in core [19:49:19] API changes maybe? [19:49:28] RelEng changes to browser test infrastructure? [19:49:33] Would have happened within the last 24hrs [19:52:41] hashar: https://gerrit.wikimedia.org/r/#/c/integration/config/+/500568/ maybe? [19:59:44] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Browser-Tests, 10Patch-For-Review, 10Ruby: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10Jdlrobson) The "I am logged into the mobile website" step seems to be defined inside the mediawiki selenium gem a... [20:03:26] Hey folks. Do you know if test.wikipedia.org just got rebooted? [20:04:21] We're seeing some errors in ORES that I think are due to test.wikipedia.org being offline briefly. [20:05:31] So is the ! in my phab notifications known? (sorry if this was mentioned like a dozen times before now) [20:05:58] 'unresolved setup issues' [20:10:41] halfak: rebooted? it just got the new version of MW [20:10:49] https://tools.wmflabs.org/sal [20:11:17] aha! We use testwiki for some of our icinga tests. Seems like we should stop that. [20:11:30] there should be no "reboot" though [20:11:39] halfak: it behaves just like the other wikis when a deployment happens [20:11:41] Yeah. Sorry about my non-specific language [20:11:52] Right. But gets updates more often, right? [20:12:12] halfak: ish, sometimes people do hotfixes there to test things, but not very often. [20:12:25] otherwise it just gets the normal train update (so on tuesdays) [20:12:35] robh: where? [20:12:42] Oh wait. Does enwiki become unavailable for a few minutes when it gets updated? [20:13:19] no, that's what I mean, there is no unavailability caused by our deployments :) [20:13:31] greg-g: when i login to phab [20:13:34] I mean, unless someone committed broken code ;) [20:13:40] oh, its gone now [20:13:45] it was totally there when i joined! [20:13:51] suuuure [20:13:58] no idea what you're talking about robh :) [20:14:06] if it cannot be reproduced it didnt happen. [20:14:48] greg-g, gotcha. But test.wikipedia.org was definitely unavailable for a solid minute a moment ago. [20:14:49] robh: sure it wasn't jenkins? it sometimes gives that to users with special permissions [20:15:03] what do you mean? [20:15:07] halfak: huh, I'd check -operations for anything interesting/related [20:15:10] it was in phab, was a small ! next to notifications [20:15:16] i didnt screen shot [20:15:26] it went away just now when i refreshed [20:15:32] greg-g, When ORES reported it couldn't talk to test.wikipedia.org, I checked my browser and couldn't load a page either. [20:15:54] so the icons are like wmf logo, phabricator, notifications, coherence/chat icon, then i had a yellow ! [20:15:58] Anyway, I think I know how to get around this on our side so we don't get a bunch of icinga noise :) [20:16:03] had not seen that, but meh its gone so bad caching during update perhaps? [20:16:09] robh: https://people.wikimedia.org/~gjg/tmp/Selection_052.png [20:16:15] nope [20:16:31] halfak: timestamps-ish? [20:18:33] ORES first complained at 20:00:16 UTC (via icinga) [20:18:54] I checked test.wikipedia.org at approximately 20:03 UTC [20:20:20] I don't see anything obvious in -operations around that time (so production icinga things) [20:21:22] Hmm. Does ORES icinga not post to -operations? You can see the messages I'm talking about in -ai. [20:23:50] not in there anymore :) I cut down my IRC channel membership drastically recently [20:24:17] is it the same production icinga host? if so it'd be just a config. [20:24:25] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10Wikimedia-Incident: Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10jcrespo) For example https://www.mediawiki.org/wiki/How_to_report_a_bug is a very... [20:24:57] https://wm-bot.wmflabs.org/browser/index.php?start=04%2F02%2F2019&end=04%2F02%2F2019&display=%23wikimedia-ai for reference [20:25:52] https://phabricator.wikimedia.org/T219930 [20:26:15] ^ My notes. We'll be looking into changing our tests so that fluctuations in availability of testwiki don't cause us to wake up at night :) [20:27:03] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Browser-Tests, 10Patch-For-Review, 10Ruby: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10Jdlrobson) [20:28:28] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Browser-Tests, 10Patch-For-Review, and 2 others: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10greg) [20:29:01] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Browser-Tests, 10Patch-For-Review, and 2 others: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10greg) >>! In T219920#5079196, @Jdlrobson wrote: > The "I am logged into the mobile website" step seems to be... [21:06:56] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 5.56% of data above the critical threshold [3.0] [21:24:30] 10Release-Engineering-Team (Watching / External), 10Scap, 10ORES, 10Operations, and 2 others: [Epic] ORES should use a git large file plugin for storing serialized binaries - https://phabricator.wikimedia.org/T171619 (10Halfak) [21:30:28] Niharika: re. PageAssessments - could it be a missing (') on the script? [21:30:42] if it is a sytax error it is surely failing on production as well [21:33:10] hauskatze: for T219935 it's that there's no projects in beta [21:33:11] T219935: extensions/PageAssessments/maintenance/purgeUnusedProjects.php is causing fatals on Beta - https://phabricator.wikimedia.org/T219935 [21:33:17] $usedProjectIds = array_unique( array_merge( $projectIds1, $projectIds2 ) ); [21:33:24] $usedProjectIds ends up as an empty array [21:33:35] 10Scap, 10Scoring-platform-team (Research): Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281 (10Halfak) [21:33:43] So the maintenance script should probably just guard against that, if !count( $usedProjectIds ) bail out [21:34:43] https://github.com/wikimedia/puppet/blob/production/modules/mediawiki/manifests/maintenance/pageassessments.pp <-- no logs saved for this on prod [21:35:32] Reedy: but "Error: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '))' at line 1" [21:35:42] it says 'syntax error' not project not found [21:35:50] the script didn't even run ii [21:35:52] Yes [21:35:58] 'DELETE FROM `page_assessments_projects` WHERE (pap_project_id NOT IN ())' [21:36:01] That's not valid SQL [21:36:08] you can't have an IN() with nothing in the brackets [21:36:14] aye [21:36:14] It did try and run it [21:36:19] MW doesn't do SQL validation [21:36:55] so it's $dbr->makeList( $usedProjectIds ) ? [21:37:07] Yes [21:37:15] $usedProjectIds contains no items [21:37:20] then should be fixed in the script I guess [21:37:30] [22:33:36] So the maintenance script should probably just guard against that, if !count( $usedProjectIds ) bail out [21:37:45] ack - but I don't know how to do it :) [21:38:56] aezell is taking a look Reedy (thanks!) [21:39:21] I'm guessing on whichever beta wiki no one has created any projects [21:39:42] also test.wikipedia.beta.wmflabs.org doesn't seem to exist but well [21:39:46] Reedy, you and are thinking it's the same problem. [21:39:51] I'm taking a look now. [21:40:05] (03PS3) 10Krinkle: jjb: Updated Fresnel to 0.2.2 [integration/config] - 10https://gerrit.wikimedia.org/r/500511 (owner: 10Phedenskog) [21:41:42] (03CR) 10Krinkle: "Thanks Hashar." [integration/config] - 10https://gerrit.wikimedia.org/r/500511 (owner: 10Phedenskog) [21:47:29] Reedy: also, not sure if any of this would be relevant too: https://kibana4.wmflabs.org/goto/acd4fcdf94abe8f14e6f4d6051642607 [21:51:59] Here's a patch for those interested: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/PageAssessments/+/500850 [22:06:31] 10Release-Engineering-Team, 10Readers-Web-Backlog, 10Browser-Tests, 10Patch-For-Review, and 2 others: CI failing: Steps that require login are failing - https://phabricator.wikimedia.org/T219920 (10Jdlrobson) From what I can see the Ruby log_in method no longer works. Maybe there was an api/security fix?... [22:10:59] aezell: thanks - it's being deployed to beta atm [22:11:18] hauskatze: You're welcome. Let me know if the errors don't clear up. [22:12:36] aezell: sure, once it's there I can try and run it manually, see what the output is [22:12:47] Thanks. [22:24:23] !log maurelio@deployment-deploy01:~$ mwscript extensions/PageAssessments/maintenance/purgeUnusedProjects.php --wiki=enwikivoyage | T219935 [22:24:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:24:30] T219935: extensions/PageAssessments/maintenance/purgeUnusedProjects.php is causing fatals on Beta - https://phabricator.wikimedia.org/T219935 [22:29:43] aezell: tested - works [22:29:46] ty! [22:30:00] hauskatze: You're welcome. Thanks for validating it. [22:30:31] now I can go to sleep (after I finish some income tax stuff) [23:37:48] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10RobH) a:03greg