[00:02:20] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 4.49% of data above the critical threshold [3.0] [00:25:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Jenkins: phpunit drops dead on some extension tests on hhvm - https://phabricator.wikimedia.org/T217384 (10Smalyshev) [00:25:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10HHVM, 10Jenkins: phpunit drops dead on some extension tests on hhvm - https://phabricator.wikimedia.org/T217384 (10Smalyshev) [00:25:42] Project beta-scap-eqiad build #241190: 04FAILURE in 3.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241190/ [00:26:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10HHVM, 10Jenkins: phpunit drops dead on some extension tests on hhvm - https://phabricator.wikimedia.org/T217384 (10Smalyshev) Same as {T216689} ? [00:28:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10HHVM, 10Jenkins: phpunit drops dead on some extension tests on hhvm - https://phabricator.wikimedia.org/T217384 (10Smalyshev) [00:28:42] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10Smalyshev) [00:31:23] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10Smalyshev) > Su... [00:44:19] Yippee, build fixed! [00:44:20] Project beta-scap-eqiad build #241191: 09FIXED in 9 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241191/ [01:32:21] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [2.0] [01:38:10] (03CR) 10MarkAHershberger: Start work on docker-driven make to produce tarballs (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/495160 (owner: 10MarkAHershberger) [01:45:47] (03PS1) 10MarkAHershberger: Add some better argument handling [integration/config] - 10https://gerrit.wikimedia.org/r/496085 [01:47:19] (03CR) 10jerkins-bot: [V: 04-1] Add some better argument handling [integration/config] - 10https://gerrit.wikimedia.org/r/496085 (owner: 10MarkAHershberger) [02:32:25] 10Beta-Cluster-Infrastructure, 10Elasticsearch, 10Wikimedia-Logstash, 10Discovery-Search (Current work): ApiFeatureUsage data is not being populated in the Beta Cluster - https://phabricator.wikimedia.org/T183156 (10EBernhardson) After looking in horizon, I realized this is actually configured from puppet.... [06:23:57] 10Release-Engineering-Team, 10ChangeProp, 10WMF-JobQueue, 10Core Platform Team Backlog (Watching / External), and 2 others: Delete the mediawiki/services/cp-jobqueue repo - https://phabricator.wikimedia.org/T217683 (10greg) 05Open→03Resolved a:03greg Marked as hidden (there is no delete). [08:35:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10hashar) > This does not belong in DevelopmentSettings.php.... [08:53:46] and I found a bug in quibble :/ [09:04:01] (03PS1) 10Hashar: (WIP WIP WIP WIP) Better arg handling [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 [09:04:08] 10Continuous-Integration-Infrastructure, 10Shinken: Host DOWN alert for integration-publishing02! - https://phabricator.wikimedia.org/T218146 (10Peachey88) [09:04:59] gehel: I could not deploy the java8 jenkins job update yesterday, I had a board meeting :/ [09:05:04] but I guess I can do that now [09:05:30] hashar: Tyler took care of it, we're now all good! [09:05:35] ah perfect! [09:05:40] team work :))))))))))) [09:05:46] until I find another problem that needs fixing :) [09:05:49] and yeah I should have tested the containers :/ [09:06:16] I should have as well, but I barely understand the dependencies between all the pieces [09:06:27] but that's improving! [09:06:30] I don't either [09:06:49] but we can chat about it and figure it out (conf call works for me) [09:07:49] nah, atm it works well enough and my understanding is improving [09:50:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10hashar) Anyway I went to reproduce on my local machine to... [10:01:55] (03PS1) 10Hashar: jjb: raises quibble db tmpfs from 256M to 320M [integration/config] - 10https://gerrit.wikimedia.org/r/496134 (https://phabricator.wikimedia.org/T217654) [10:03:11] !log Bump Quibble tmpfs disk space used to hold the database from 256MBytes to 320Mbytes. l10n_cache causes an overflow # T217654 [10:03:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:03:14] T217654: Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 [10:09:31] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 173" [integration/config] - 10https://gerrit.wikimedia.org/r/496134 (https://phabricator.wikimedia.org/T217654) (owner: 10Hashar) [10:11:52] (03Merged) 10jenkins-bot: jjb: raises quibble db tmpfs from 256M to 320M [integration/config] - 10https://gerrit.wikimedia.org/r/496134 (https://phabricator.wikimedia.org/T217654) (owner: 10Hashar) [10:14:02] (03CR) 10Alexandros Kosiaris: "The base image is per https://phabricator.wikimedia.org/T205911#4982711 unfortunately a dead end. With all the ways that nodejs would disc" [blubber] - 10https://gerrit.wikimedia.org/r/492922 (https://phabricator.wikimedia.org/T205911) (owner: 10Alexandros Kosiaris) [10:18:25] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10zeljkofilipin) [10:18:27] 10Release-Engineering-Team (Kanban): Check gerrithub's info in spreadsheet - https://phabricator.wikimedia.org/T217890 (10zeljkofilipin) 05Open→03Resolved [10:33:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10hashar) p:05Unbreak!→03High The... [10:39:42] !log Triggered tests for ContentTranslation changes that had label:verified=-1 # T217654 [10:39:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:39:46] T217654: Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 [10:46:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10hashar) a:03hashar [11:01:29] Project beta-scap-eqiad build #241246: 04FAILURE in 8 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241246/ [11:08:43] down for everyone or just me? integration.wikimedia.org [11:12:35] Yippee, build fixed! [11:12:36] Project beta-scap-eqiad build #241247: 09FIXED in 9 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241247/ [11:30:33] mutante: works for me [11:32:06] zeljkof: thanks, it does for me as well now [11:33:06] 10Phabricator (Upstream), 10Upstream: Burnup rate month date shown in tool tip is off by one. January is month zero! - https://phabricator.wikimedia.org/T164478 (10epriestley) See , although we're moving toward , as noted above. [11:38:18] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) [11:44:41] 10Diffusion, 10Phabricator (Upstream), 10Upstream: File contents do not show up if content is not valid utf-8 - https://phabricator.wikimedia.org/T144672 (10epriestley) I can no longer reproduce this. The file is now automatically detected as SHIFT-JIS and converted on display, which is probably incorrect,... [11:48:01] 10Phabricator (Upstream), 10Upstream: Paste search shows wrong length - https://phabricator.wikimedia.org/T140324 (10epriestley) After (upstream version of the same change above) I can't reproduce this, and spot-checking `/paste/` on this install doesn't show any obvious... [11:59:49] 10Phabricator (Upstream), 10Upstream: Allow overwriting/changing saved queries - https://phabricator.wikimedia.org/T113556 (10epriestley) I'm going to retitle this to align with the upstream task since I everyone is actually talking about a reasonable feature ("Make it easier to change which query a dashboard... [12:00:13] 10Phabricator (Upstream), 10Upstream: Make it easier to update the query a Phabricator Dashboard "Query" Panel uses - https://phabricator.wikimedia.org/T113556 (10epriestley) [12:03:07] 10Phabricator (Upstream), 10Upstream: Phantom unread notifications when users are subscribed to objects they are not allowed to view - https://phabricator.wikimedia.org/T102653 (10epriestley) I believe this was (almost entirely) resolved by . [12:09:37] 10Phabricator (Upstream), 10Upstream: Task created via Conduit is reported in mail as "Reopened" - https://phabricator.wikimedia.org/T88005 (10epriestley) I'm fairly sure this is fixed for `maniphest.edit` (the more modern, preferred API). The older `maniphest.createtask` may still have some weird behavior, bu... [12:22:29] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10hashar) From T204153: I have created a dummy test that tracks whether dependencies of gated extensions are also in the gate https://gerrit.wik... [13:06:21] 10Phabricator (Upstream), 10Upstream: Add another way to add two factor auth - https://phabricator.wikimedia.org/T187256 (10epriestley) Phabricator now supports SMS MFA, although we strongly discourage using it because it has a long history of being compromised by attackers (see 10Phabricator (Upstream), 10Upstream: Phabricator says Asia/Kolkata timezone is UTC +5 but it's not - https://phabricator.wikimedia.org/T185213 (10epriestley) Likely resolved by . [13:23:08] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675 (10zeljkofilipin) 1.33.0-wmf.21 at group 1 ([[ https://tools.wmflabs.org/sal/log/AWl3MAVFIm9Dp5A3Z2oZ | sal ]], [[ https://gerrit.wikimedi... [13:23:45] 10Phabricator (Upstream), 10Upstream: Add another way to add two factor auth - https://phabricator.wikimedia.org/T187256 (10TheDJ) @epriestley we have similar challenges with MediaWiki. Esp. in the context of Wikipedia, where we simply have too little info to easily do recovery if a 2FA key is lost (which happ... [13:24:42] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 (10hashar) I have... [13:25:56] (03PS1) 10Hashar: Revert "jjb: Switch MW+Parsoid jobs on hhvm to Stretch (quibble-stretch-hhvm)" [integration/config] - 10https://gerrit.wikimedia.org/r/496162 [13:30:30] 10Phabricator (Upstream), 10Upstream: "Edit Paths" in Owners interface is unusable: Lists all repositories in an unsorted select box with no search - https://phabricator.wikimedia.org/T140713 (10epriestley) Likely resolved by . [13:33:17] 10Phabricator (Upstream), 10Upstream: Moving tasks between columns on workboard changed task priority (when sorting tasks by priority) - https://phabricator.wikimedia.org/T136400 (10epriestley) Likely resolved in the upcoming 2019 Week 11 release, see and related tasks. [13:36:37] (03PS2) 10Hashar: Revert "jjb: Switch MW+Parsoid jobs on hhvm to Stretch (quibble-stretch-hhvm)" [integration/config] - 10https://gerrit.wikimedia.org/r/496162 (https://phabricator.wikimedia.org/T216689) [13:41:38] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689 [13:41:43] (03CR) 10Hashar: [C: 03+2] "I have updated the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/496162 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [13:41:51] 10Phabricator (Upstream), 10Upstream: Support "pinning" tasks to the top of a column - https://phabricator.wikimedia.org/T103427 (10epriestley) This is possibly resolved (or, at least, improved) by and related work. Within a priority group (say, "High"), you now get beh... [13:44:07] (03Merged) 10jenkins-bot: Revert "jjb: Switch MW+Parsoid jobs on hhvm to Stretch (quibble-stretch-hhvm)" [integration/config] - 10https://gerrit.wikimedia.org/r/496162 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [13:47:15] (03CR) 10Thcipriani: maintenance: Cleanup old Docker images at a lower threshold (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/490505 (https://phabricator.wikimedia.org/T177867) (owner: 10Dduvall) [13:51:13] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689 [13:51:22] 10Phabricator (Upstream), 10Upstream: Public log for Phabricator user account role change (e.g. deactivation, admin rights) - https://phabricator.wikimedia.org/T102576 (10epriestley) Upstream changes in and mean that account disabl... [14:00:28] (03CR) 10Krinkle: "Per https://gerrit.wikimedia.org/r/#/c/integration/config/+/489434/ , it says:" [integration/config] - 10https://gerrit.wikimedia.org/r/496162 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [14:00:50] (03PS1) 10Gopavasanth: Edit Project Config [labs/tools/VideoCutTool] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/496167 [14:03:12] 10Continuous-Integration-Config: Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10Nikerabbit) [14:04:09] (03PS1) 10Hashar: docker: move HHVM core dump report to workspace [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) [14:04:41] (03CR) 10Krinkle: "aside from Chrome Headless, it also resurface $HOME nonwriteable bug: https://phabricator.wikimedia.org/T218209" [integration/config] - 10https://gerrit.wikimedia.org/r/496162 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [14:05:19] 10Continuous-Integration-Config, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10Krinkle) Presumably due to which was just depl... [14:06:44] (03PS2) 10Hashar: docker: move HHVM core dump report to workspace [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) [14:09:48] 10Continuous-Integration-Config, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10Nikerabbit) [14:18:25] (03PS2) 10Thcipriani: post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 [14:19:38] (03CR) 10Thcipriani: post pipeline: fix ref filter (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (owner: 10Thcipriani) [15:07:52] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find more CI candidates, divide list based on very hard requirements - https://phabricator.wikimedia.org/T217593 (10zeljkofilipin) After reviewing a few lists of tools (see [[ https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures... [15:08:14] 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10zeljkofilipin) [15:08:17] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find more CI candidates, divide list based on very hard requirements - https://phabricator.wikimedia.org/T217593 (10zeljkofilipin) 05Open→03Resolved [15:13:15] i'd need some dpeloyment-prep advice [15:13:47] we need to set up cassandra on one node, but we don't have a role for it (nor should it exist), we only have a profile, but applying it doesn't work [15:14:18] so the only thing i can think of is to create a fake role::cassandra in ops/puppet and cherry-pick it on the puppetmaster03, but that's ugly to say the least [15:14:23] any better ideas? [15:17:44] (03CR) 10Thcipriani: Add some better argument handling (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/496085 (owner: 10MarkAHershberger) [15:22:21] hrm, why doesn't applying the profile work? [15:36:01] I'm really looking forward to the next time we upgrade Phabricator, given that Evan has just done a bunch of improvements that I filed back in 2015 around task sorting/grouping on workboards [15:40:58] 10Continuous-Integration-Config, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10Lucas_Werkmeister_WMDE) p:05Triage→03Unbreak! Merge blocker means UBN, as far as I’m aware. (It’s blocking m... [15:42:46] (03CR) 10Subramanya Sastry: "*ping* is this ready to go?" [integration/config] - 10https://gerrit.wikimedia.org/r/494802 (https://phabricator.wikimedia.org/T216102) (owner: 10C. Scott Ananian) [15:45:42] 10Continuous-Integration-Config, 10MediaWiki-extensions-JsonConfig, 10Reading-Infrastructure-Team-Backlog, 10Maps (Kartographer), and 2 others: Move JCMapDataContent from JsonConfig to Kartographer - https://phabricator.wikimedia.org/T163274 (10Jhernandez) p:05Triage→03Normal [15:46:11] 10Continuous-Integration-Config, 10MediaWiki-extensions-JsonConfig, 10Reading-Infrastructure-Team-Backlog, 10Maps (Kartographer), and 2 others: Move JCMapDataContent from JsonConfig to Kartographer - https://phabricator.wikimedia.org/T163274 (10Jhernandez) 05Open→03Stalled [15:48:08] (03CR) 10Jforrester: "So… we should revert, right? Occasional quibble fatals are irritating, but better than continuous fatals." [integration/config] - 10https://gerrit.wikimedia.org/r/496162 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [15:50:29] (03CR) 10Thcipriani: Start work on docker-driven make to produce tarballs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/495160 (owner: 10MarkAHershberger) [15:53:27] thcipriani: i don't think applying profiles ever worked [15:55:22] mobrovac: FWIW, we do have the "beta" role. You could do: role::beta::cassandra [15:55:35] ah! [15:55:38] good idea thcipriani [15:55:40] it's been used in hacky situations like this before :) [15:55:48] thnak you! [15:55:56] sure thing :) [16:14:49] (03PS1) 10Vedmaka Wakalaka: Adds Vedmaka Wakalaka to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/496196 [16:22:23] (03CR) 10Vedmaka Wakalaka: "I am currently working on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/495832 and going to work on some more tickets, so it would be " [integration/config] - 10https://gerrit.wikimedia.org/r/496196 (owner: 10Vedmaka Wakalaka) [16:27:22] 10Continuous-Integration-Config, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10D3r1ck01) Jenkins builds seems to be failing too, could this be related. Actually, it's for all patches I think? [16:33:26] PROBLEM - Puppet errors on deployment-sessionstore01 is CRITICAL: CRITICAL: 4.44% of data above the critical threshold [3.0] [17:11:49] Hey there! I'd like to be included into CI whitelist https://gerrit.wikimedia.org/r/c/integration/config/+/496196 , it would help me a lot with my work (https://phabricator.wikimedia.org/T208768 and https://phabricator.wikimedia.org/T208776) is there anything I need to do to achieve this ? [17:16:19] 10Continuous-Integration-Infrastructure, 10MediaWiki-Documentation: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10Krinkle) [17:16:25] 10Continuous-Integration-Infrastructure, 10MediaWiki-Documentation, 10Regression: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10Krinkle) [17:22:06] (03PS1) 10Thcipriani: Revert "Revert "jjb: Switch MW+Parsoid jobs on hhvm to Stretch (quibble-stretch-hhvm)"" [integration/config] - 10https://gerrit.wikimedia.org/r/496207 [17:23:49] (03PS2) 10Thcipriani: Revert "Revert "jjb: Switch MW+Parsoid jobs on hhvm to Stretch (quibble-stretch-hhvm)"" [integration/config] - 10https://gerrit.wikimedia.org/r/496207 (https://phabricator.wikimedia.org/T218209) [17:36:04] (03CR) 10Thcipriani: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/496207 (https://phabricator.wikimedia.org/T218209) (owner: 10Thcipriani) [17:37:13] 10Continuous-Integration-Config, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10thcipriani) >>! In T218209#5020861, @Krinkle wrote: > Presumably due to (03Merged) 10jenkins-bot: Revert "Revert "jjb: Switch MW+Parsoid jobs on hhvm to Stretch (quibble-stretch-hhvm)"" [integration/config] - 10https://gerrit.wikimedia.org/r/496207 (https://phabricator.wikimedia.org/T218209) (owner: 10Thcipriani) [17:46:18] 10Continuous-Integration-Config, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10thcipriani) p:05Unbreak!→03High > https://integration.wikimedia.org/ci/job/wmf-quibble... [17:50:43] ;( [17:50:50] and now I hate docker even more [18:01:43] (03PS5) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [18:08:21] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [18:08:23] (03CR) 10jerkins-bot: [V: 04-1] Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [18:12:39] (03PS3) 10Hashar: docker: move HHVM core dump report to workspace [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) [18:14:24] is there someone here who understands packagist? [18:14:55] 10Continuous-Integration-Config, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: Failed to create /nonexistent/.pki/nssdb directory - https://phabricator.wikimedia.org/T218209 (10hashar) 05Open→03Resolved a:03thcipriani Just a note: [ERROR:nss_util.cc(94)]... [18:15:03] there seems to be a "wikimedia" user (*and* a "mediawiki" user) on packagist, who has the permission bits for those? [18:15:19] i'd like to publish a new package, wikimedia/zest [18:18:27] RECOVERY - Puppet errors on deployment-sessionstore01 is OK: OK: Less than 1.00% above the threshold [2.0] [18:18:35] (03PS4) 10Hashar: docker: move HHVM core dump report to workspace [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) [18:21:54] thcipriani: got some random review for you https://gerrit.wikimedia.org/r/#/c/integration/config/+/496168/ :) [18:22:10] thcipriani: my intent is to have hhvm to log some report when a segfault occurs :/ [18:22:28] hashar: makes sense, I'll give it a look [18:23:16] thcipriani: and thank you for the revert earlier [18:24:15] hashar: yw, glad it solved the issues folks were having [18:24:31] yeah [18:24:34] the mess is going to be https://phabricator.wikimedia.org/T216689 [18:24:39] and [18:24:46] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689 [18:25:03] enabling cores is probably fine [18:25:12] but I have no idea what kind of new packages are going to be added :/ [18:25:15] so [18:25:18] I gotta build it locally [18:25:29] then do a diff :-( [18:26:03] ugh [18:26:17] running apt update doing the build is hmm [18:26:21] not ideal [18:26:26] (I am trying to stay polite there) [18:27:14] :) [18:27:42] my hacking life is miserable [18:27:44] I hate softwares [18:28:15] * hauskatze gives a piece of brioche to hashar to make him a bit happier [18:28:31] oh actually hmm brioche is interesting ;) [18:28:37] thcipriani: real quick check, no deployment coming up in 30 mins right? [18:28:47] can Pchelolo and I enable our logging on group1 wikis ? [18:28:49] now [18:29:46] ottomata: there is no train deployment happening in 30 minutes, that's correct. You should be able to deploy a patch if you need to. [18:29:52] hauskatze: that is half a recomfort thank you ;) half is because I love brioche, half because in 1788 or 1789 when people were protesting about the ongoing starving, they were claiming bread. To which the queen might have replied: "they are hungry? Lets give them brioche" :D [18:29:59] hauskatze: tldr: thanks! [18:30:25] france had a queen? [18:30:44] the wife of Louis XVI éme [18:31:00] oh, so she got the title from him. [18:31:12] not sure, she never reigned [18:31:21] Queen consort I guess [18:31:42] I dont think, we make the difference between queen and queen consort [18:32:15] hmm, Clovis I, nice name :) [18:32:20] oh [18:33:23] actually women were banned from reigning [18:33:31] except for interim purposes (régence) [18:33:41] Loi salique I guess [18:33:43] same here [18:34:21] danke thcipriani [18:34:55] hashar hauskatze not here :P [18:35:19] paladox: blame brioche [18:35:26] lol [18:40:16] 10Beta-Cluster-Infrastructure, 10Elasticsearch, 10Wikimedia-Logstash, 10Discovery-Search (Current work), 10Patch-For-Review: ApiFeatureUsage data is not being populated in the Beta Cluster - https://phabricator.wikimedia.org/T183156 (10EBernhardson) due to unrelated issues the restart to pickup the confi... [18:40:54] (03CR) 10Hashar: "Build locally and I did a diff of dpkg -l with the previous versions of the images. The only difference is for jessie:" [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [18:41:21] thcipriani: I did the dpkg -l diff there is no meaningful difference. Just a security update for jessie [18:42:33] nice [18:43:03] and I could not find a way to trigger a core dump to verify the report is actually generated :/ [18:43:24] also [18:43:39] I have let my laptop run ContentTranslation tests in a loop for a few hours while I was out [18:43:45] but .. it did not crash ;( [18:44:09] (03CR) 10Thcipriani: [C: 03+2] docker: move HHVM core dump report to workspace [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [18:44:17] whatever happens must be over my pay grade (or I need a raise to become smarter?) [18:44:33] thx [18:44:35] will rebuild it [18:44:48] cool, looks like a useful change for troubleshooting. [18:44:54] yeah [18:45:01] hopefully it helps to suss out the problem. [18:45:11] fun thing is the file had a comment in that sense [18:45:40] (03Merged) 10jenkins-bot: docker: move HHVM core dump report to workspace [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [18:46:21] rebuilding [18:46:31] (which really should be made an automatic process) [18:48:37] !log Building containers releng/quibble-jessie-hhvm and releng/quibble-stretch-hhvm with HHVM core_dump_report enabled # T216689 [18:48:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:48:39] T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11) - https://phabricator.wikimedia.org/T216689 [18:56:08] (03PS1) 10Hashar: Enable HHVM crash report for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/496239 (https://phabricator.wikimedia.org/T216689) [18:56:24] (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/496239 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [18:59:37] (03Merged) 10jenkins-bot: Enable HHVM crash report for Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/496239 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [19:00:51] 10Continuous-Integration-Infrastructure: integration-slave-docker-1046 is offline - https://phabricator.wikimedia.org/T218245 (10hashar) [19:03:33] 10Continuous-Integration-Infrastructure: integration-slave-docker-1046 is offline - https://phabricator.wikimedia.org/T218245 (10hashar) From SAL: **2019-02-13** 21:21 bringing integration-slave-docker-1046 and integration-slave-jessie-1001 back online 21:15 removing old docker im... [19:04:24] !log hard rebooting integration-slave-docker-1046 , not reachable over ssh # T218245 [19:04:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:04:27] T218245: integration-slave-docker-1046 is offline - https://phabricator.wikimedia.org/T218245 [19:05:32] RECOVERY - Host integration-slave-docker-1046 is UP: PING OK - Packet loss = 0%, RTA = 0.59 ms [19:08:02] fun fact [19:08:06] ffmpeg segfault as well ... [19:13:09] !log integration-slave-docker-1046 is back online # T218245 [19:13:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:13:12] T218245: integration-slave-docker-1046 is offline - https://phabricator.wikimedia.org/T218245 [19:13:17] 10Continuous-Integration-Infrastructure: integration-slave-docker-1046 is offline - https://phabricator.wikimedia.org/T218245 (10hashar) 05Open→03Resolved a:03hashar [19:14:25] ... [19:14:33] I have been too enthusiast [19:15:55] PROBLEM - Puppet staleness on integration-slave-docker-1046 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [19:20:54] RECOVERY - Puppet staleness on integration-slave-docker-1046 is OK: OK: Less than 1.00% above the threshold [3600.0] [19:26:55] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689 [19:54:04] Krenair: do you know about a local hack "Add puppetdb passwords" on the puppet private repo on deployment-puppetmaster03 which does not rebase cleanly? [19:54:18] if you have time to check at some point that would be nice! [19:54:22] Thansk! [19:54:51] I know about it [19:55:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker - https://phabricator.wikimedia.org/T217654 (10hashar) 05Open→03Resolve... [19:55:21] gehel, um the rebase is just dealing with multiple things at the bottom of a hiera file [19:55:26] doesn't need any commit-specific knowledge [19:55:56] fixed [19:56:01] thanks! [19:56:30] I can move that part further up so people are less likely to conflict with it [19:57:42] done [19:57:51] even better! thanks again [19:59:08] 10Continuous-Integration-Infrastructure, 10Operations, 10Packaging, 10Patch-For-Review: Upgrade jenkins-debian-glue to v0.20.0 - https://phabricator.wikimedia.org/T212774 (10hashar) > As far as i can tell this only needs to go to contint2001.wikimedia.org and contint1001.wikimedia.org Those run the CI st... [20:13:39] !log (beta): Update mobileapps to 5865552 (7074964 d6dc3cd fbc6262) [20:13:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:25:53] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689 [20:26:08] how does beta deployment of new extensions work these days when there is no separate extension list for beta? Do wmf.X branches need to exist by the time the extension is added to wmf-config/extension-list, or only the mediawiki/extensions submodule? [20:33:46] 10Continuous-Integration-Config, 10Lexicographical data, 10Wikidata, 10MW-1.33-notes (1.33.0-wmf.21; 2019-03-12), and 4 others: Enable phan checks for WikibaseLexeme extension - https://phabricator.wikimedia.org/T215556 (10Smalyshev) [20:37:42] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Find way to exclude php 5.4 files from vendor lint task - https://phabricator.wikimedia.org/T170641 (10Eileenmcnaughton) 05Open→03Resolved a:03Eileenmcnaughton no more php 5.4 - closing [20:37:53] thcipriani: any ideas about https://phabricator.wikimedia.org/T218253 ? [20:38:45] !log Added integration-slave-docker-1045 to Jenkins. The instance existed in WMCS but was not in Jenkins [20:38:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:41:22] is phanchecker playing up for anyone else? [20:41:40] Seeing lots of false positives such as https://integration.wikimedia.org/ci/job/mwext-php70-phan-seccheck-docker/43428/console [20:43:34] kostajh: hrm, looking [20:43:39] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689 [20:44:21] !log marking integration-slave-docker-1046 offline [20:44:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:48:21] thcipriani: I have added 1046 a few minutes ago ( https://phabricator.wikimedia.org/T218245 ) feel free to just delete it? [20:48:51] I failed to notice it might have been affected by the WMCS disk outage from a few weeks ago [20:48:58] so it is probably corrupted and irrecoverable [20:49:01] my bad :/ [20:49:21] now. I am going to have some light dinner and head to bed *wave* [20:50:04] ah, that would make sense, I have not seen this error before, I will delete and add a new one. [20:53:00] jdlrobson: Yes that's T218253 [20:53:01] T218253: docker: failed to register layer: error creating overlay mount to - https://phabricator.wikimedia.org/T218253 [20:55:45] RoanKattouw: jdlrobson that one should be solved when I took integration-slave-docker-1046 out of the ci pool. Corrupt disk. I'm deleting now and then I'll close that task. [20:58:05] !log deleting bigram CI instance integration-slave-docker-1046 due to corrupt disk cf: T218245 [20:58:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:58:09] T218245: integration-slave-docker-1046 is offline - https://phabricator.wikimedia.org/T218245 [21:03:27] PROBLEM - Host integration-slave-docker-1046 is DOWN: CRITICAL - Host Unreachable (172.16.1.115) [21:11:44] thcipriani: thanks for sorting that out! [21:12:16] kostajh: yw, thanks for spotting it and filing a task [21:26:58] !log pool new bigram CI instance integration-slave-docker-1054 [21:26:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:14] 10Deployments, 10Release-Engineering-Team (Kanban), 10MediaWiki-Internationalization, 10Wikimedia-Incident, 10Wikimedia-production-error: Post-mortem "MWException: No localisation cache found for English." - https://phabricator.wikimedia.org/T217719 (10greg) So, I guess the question is: What happens when... [21:50:54] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10Elasticsearch, 10Beta-Cluster-reproducible: Search error on beta meta - https://phabricator.wikimedia.org/T218256 (10Tgr) Looks like beta ES is down, [[https://logstash-beta.wmflabs.org/goto/afe9f951e0d5f2e6f7e698522445a428|the log]] is full of `502 Bad... [21:54:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Jenkins: Jenkins warning JENKINS-2111 path sanitization ineffective - https://phabricator.wikimedia.org/T217791 (10greg) @hashar severity/priority? [21:56:09] 10Continuous-Integration-Infrastructure, 10HHVM, 10Language-Team (Language-2019-January-March), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11... - https://phabricator.wikimedia.org/T216689 [21:56:56] (03CR) 10Hashar: "kill -SIGSEGV $(pidof php) hhvm yields:" [integration/config] - 10https://gerrit.wikimedia.org/r/496168 (https://phabricator.wikimedia.org/T216689) (owner: 10Hashar) [21:56:58] (03PS2) 10tosfos: Adds Vedmaka Wakalaka to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/496196 (owner: 10Vedmaka Wakalaka) [22:08:55] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10Elasticsearch, 10Beta-Cluster-reproducible: Search error on beta meta - https://phabricator.wikimedia.org/T218256 (10EBernhardson) An elasticsearch upgrade is in progress, forced a little early due to an issue that downgraded an instance for 6 to 5 (whic... [22:23:38] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10Elasticsearch, 10Beta-Cluster-reproducible: Search error on beta meta - https://phabricator.wikimedia.org/T218256 (10Tgr) 05Open→03Invalid Ah, cool, thanks. I should have looked at the logs which make it pretty clear this is transient. I just tend to... [22:30:35] paladox: any idea why [22:30:35] https://gerrit.wikimedia.org/r/changes/?pp=0&o=TRACKING_IDS&o=DETAILED_LABELS&q=bug:T210653+OR+bug:T195482+OR+bug:T212465+OR+bug:T216495+OR+bug:T210658+OR+bug:T195478+OR+bug:T217102+OR+bug:T186737+OR+bug:T213847+OR+bug:T150377+OR+bug:T218173+OR+bug:T217298+OR+bug:T215477+OR+bug:T201339+OR+bug:T217296+OR+bug:T216198+OR+bug:T204627+OR+bug:T213352+OR+bug:T217820+OR+bug:T168861+OR+bug:T139317+OR+bug:T216264+OR+bug:T2129 [22:30:35] 61+OR+bug:T208605 is returning an empty string for me? was working yesterday but not today (did anything change to Gerrit?) [22:30:42] T186737: Let me choose whether to present a survey to logged-in or logged-out editors - https://phabricator.wikimedia.org/T186737 [22:30:43] T150377: Use 'outside' instead of 'Inside' for list-style in Mobile version so that long line bullet list items are more scannable - https://phabricator.wikimedia.org/T150377 [22:30:43] T215477: Tag Thanks actions with AMC tag - https://phabricator.wikimedia.org/T215477 [22:30:44] T216264: Deprecate 'View.$' - https://phabricator.wikimedia.org/T216264 [22:30:44] T216495: [EPIC] Diversity quicksurveys - https://phabricator.wikimedia.org/T216495 [22:30:44] * paladox looks [22:30:45] T217102: MFA: TalkSectionAddOverlay should use Overlay.make pattern - https://phabricator.wikimedia.org/T217102 [22:30:45] T216198: MFA: ImageOverlay should be an Overlay Composed of Components - https://phabricator.wikimedia.org/T216198 [22:30:45] T210658: [EPIC] AMC Navigation - page/talk toggle - https://phabricator.wikimedia.org/T210658 [22:30:46] T210653: [EPIC] AMC - Opting in - https://phabricator.wikimedia.org/T210653 [22:30:46] T195478: [EPIC] Speed up unit test execution and increase code coverage - https://phabricator.wikimedia.org/T195478 [22:30:47] lol [22:30:47] T217820: Regression: "Expand all sections" setting in Minerva is broken - https://phabricator.wikimedia.org/T217820 [22:30:47] T139317: Allow quicksurvey to target based on edit count - https://phabricator.wikimedia.org/T139317 [22:30:48] T168861: Infobox caption should be properly formatted in the mobile view - https://phabricator.wikimedia.org/T168861 [22:30:48] T195482: [EPIC] Review and refactor Minerva components - https://phabricator.wikimedia.org/T195482 [22:30:48] T218173: [Bug/Regression] New talk topic edits are lost when I click browser back button - https://phabricator.wikimedia.org/T218173 [22:30:49] T212465: [EPIC] None of our View's should exhibit 2 levels of inheritance - https://phabricator.wikimedia.org/T212465 [22:30:52] oh no [22:30:58] https://www.irccloud.com/pastebin/jiiUwrM2/ [22:31:21] jdlrobson can you paste the link in a paste? [22:31:29] Sure [22:31:33] thanks [22:31:37] https://www.irccloud.com/pastebin/GBHuc6qA/ [22:32:04] shows a json file for me [22:32:19] with a big long line [22:32:38] maybe it's the headers I'm sending? [22:32:56] are you using a python application for this? [22:33:15] ie https://github.com/dpursehouse/pygerrit2 [22:35:33] it's a chrome extension so using node fetch [22:35:50] ah ok [22:36:13] https://chrome.google.com/webstore/detail/pherrit/polcefipbgcdfkpbmmbdjgkgfgjoebij [22:36:23] it stopped working for me today/yesterday [22:36:23] content header i get is Content-Type: application/json;charset=utf-8 [22:38:02] jdlrobson note that the json that is recived has a )]}' which *could* affect the parsing. [22:38:09] i think gerrit in PolyGerrit strips it [22:38:14] yeh it has a workaround for that [22:38:24] when I debug it's getting an empty string which is bizarre [22:38:35] jdlrobson https://github.com/GerritCodeReview/gerrit/blob/master/polygerrit-ui/app/elements/shared/gr-rest-api-interface/gr-rest-api-interface.js#L405 [22:38:38] could it possibly be sending binary content now when before it wasn't? [22:38:55] Nope [22:40:18] https://github.com/j4n-co/pherrit/blob/09060e83ea303d517d152898d6f1abe0396ca8f4/src/lib/GerritResponseParser.js#L5 [22:40:23] hmm, what does reduce do [22:40:36] it's expecting an array [22:41:00] when I add console.log('g', responseText); before responseText = responseText.replace( /^.*\n/, '' ); I can see it's getting an empty string [22:41:47] paladox: did gerrit get any updates this week? [22:41:50] Nope [22:42:13] okay so it's more than likely a problem my side [22:42:19] i'll keep debugging and see what I come up with [22:42:23] curl -I -X GET < url> shows the correct content type returning [22:43:36] what about adding a console log here https://github.com/j4n-co/pherrit/blob/2049c56418271c0dd2eac2c8d54fb4c6e42c64cd/src/main.js#L17 jdlrobson [22:43:36] ? [22:43:54] for responseText [22:47:35] jdlrobson gerrit does: [22:47:35] https://github.com/GerritCodeReview/gerrit/blob/master/polygerrit-ui/app/elements/shared/gr-rest-api-interface/gr-auth.js#L73 [22:48:16] im guessing the problem is something to do with CORs but not sure why it's only just emerged as a problem [22:48:31] Yeh [22:48:44] jdlrobson does the console log say something about failing to load the url? [22:48:53] i think i had that problem [22:49:00] with zuul-status on a non http site [22:56:02] "pherrit" is an amazing name. [23:00:06] heh [23:00:26] * https [23:01:17] I want pherrit for Fx [23:03:13] I'm using https :/ not sure what happened [23:03:26] oh [23:03:53] jdlrobson anything in the console log that will tell you if it's having problems with cor? [23:04:27] heh, didn't know volvo cars used gerrit [23:04:33] no.. its a 200 with 0 bytes sent though [23:04:35] https://twitter.com/GerritReview/status/1105944745157758983 [23:04:50] then it dosen't seem to be a cors issue jdlrobson [23:05:25] 0 bytes could be either wrong header, or it's sending the url wrong [23:05:30] maybe the URL is too long [23:11:40] possibly [23:14:55] dumb question: are you still logged in? [23:15:02] /do you need to be? [23:16:17] to use that rest api you doin't need to be signed in. [23:24:30] straw grasping ¯\_(ツ)_/¯ [23:24:43] nothing in the error logs that looks related to this fwiw [23:25:13] it would likely be the url [23:26:35] the url is large but *should* still work