[00:00:37] miraheze/DiscordNotifications - Universal-Omega the build passed. [00:06:53] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 58680MiB (12% inode=99%); [00:12:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:17:33] PROBLEM - evicnotice.peakprecision.wiki - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:18:08] PROBLEM - outlaster.peakprecision.wiki - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:37:10] [02DiscordNotifications] 07Universal-Omega closed issue #73: Uploaded files approved through the Moderation extension are attributed in the feed to the approving moderator rather than the actual uploader 13https://github.com/miraheze/DiscordNotifications/issues/73 [00:37:11] [02DiscordNotifications] 07Universal-Omega commented on issue #73: Hi, please open a task on https://issue-tracker.miraheze.org/tag/discordnotifications/ instead now, thanks! 13https://github.com/miraheze/DiscordNotifications/issues/73#issuecomment-2749735901 [00:39:32] [02DiscordNotifications] 07Universal-Omega closed issue #87: Allow selection of fields you want embedded in the message 13https://github.com/miraheze/DiscordNotifications/issues/87 [00:39:33] [02DiscordNotifications] 07Universal-Omega commented on issue #87: Unfortunately I'm going to decline this. Because the full content fields don't appear in normal feeds only in the experimental CVT feature. If you still want this or I misunderstood, you can now open a task on https://issue- […] 13https://github.com/miraheze/DiscordNotifications/issues/87#issuecomment-2749738865 [00:42:23] [02DiscordNotifications] 07Universal-Omega closed issue #61: Allow catching specific log types and sending to specific webhooks, or all to the same 13https://github.com/miraheze/DiscordNotifications/issues/61 [00:42:24] [02DiscordNotifications] 07Universal-Omega commented on issue #61: Moved to https://issue-tracker.miraheze.org/T13414 13https://github.com/miraheze/DiscordNotifications/issues/61#issuecomment-2749742201 [00:44:18] [02DiscordNotifications] 07Universal-Omega closed issue #59: Add the ability to catch and monitor tiny or large edits/page creations 13https://github.com/miraheze/DiscordNotifications/issues/59 [00:44:19] [02DiscordNotifications] 07Universal-Omega commented on issue #59: Moved to https://issue-tracker.miraheze.org/T13415 13https://github.com/miraheze/DiscordNotifications/issues/59#issuecomment-2749744593 [00:45:56] [02DiscordNotifications] 07Universal-Omega closed issue #58: Add ability to catch NSFW content text sent over expermental CVT feed and wrap in spoiler tags 13https://github.com/miraheze/DiscordNotifications/issues/58 [00:45:57] [02DiscordNotifications] 07Universal-Omega commented on issue #58: Moved to https://issue-tracker.miraheze.org/T13416 13https://github.com/miraheze/DiscordNotifications/issues/58#issuecomment-2749746828 [00:47:34] [02DiscordNotifications] 07Universal-Omega closed issue #57: Add the ability to monitor page blanks and replacements in the experimental CVT feed 13https://github.com/miraheze/DiscordNotifications/issues/57 [00:47:34] [02DiscordNotifications] 07Universal-Omega commented on issue #57: Moved to https://issue-tracker.miraheze.org/T13417 13https://github.com/miraheze/DiscordNotifications/issues/57#issuecomment-2749748862 [00:49:00] [02DiscordNotifications] 07Universal-Omega closed issue #56: Add the ability to send IP edits to own expermental feed 13https://github.com/miraheze/DiscordNotifications/issues/56 [00:49:01] [02DiscordNotifications] 07Universal-Omega commented on issue #56: Moved to https://issue-tracker.miraheze.org/T13418 13https://github.com/miraheze/DiscordNotifications/issues/56#issuecomment-2749750740 [00:50:33] [02DiscordNotifications] 07Universal-Omega 04deleted 03v1.0.1 at 036e4a576 13https://github.com/miraheze/DiscordNotifications/commit/6e4a576 [00:50:38] [02DiscordNotifications] 07Universal-Omega 04deleted 03v1.0.2 at 03d67bf4d 13https://github.com/miraheze/DiscordNotifications/commit/d67bf4d [00:50:43] [02DiscordNotifications] 07Universal-Omega 04deleted 03v1.0.3 at 03a77dbb1 13https://github.com/miraheze/DiscordNotifications/commit/a77dbb1 [00:50:47] [02DiscordNotifications] 07Universal-Omega 04deleted 03v1.0.4 at 03e582a32 13https://github.com/miraheze/DiscordNotifications/commit/e582a32 [00:50:51] [02DiscordNotifications] 07Universal-Omega 04deleted 031.07 at 03469d72b 13https://github.com/miraheze/DiscordNotifications/commit/469d72b [00:50:54] [02DiscordNotifications] 07Universal-Omega 04deleted 031.10 at 03193978e 13https://github.com/miraheze/DiscordNotifications/commit/193978e [00:51:06] [02DiscordNotifications] 07Universal-Omega 04deleted 031.11 at 0394f8118 13https://github.com/miraheze/DiscordNotifications/commit/94f8118 [00:51:11] [02DiscordNotifications] 07Universal-Omega 04deleted 031.12 at 033d2f8d2 13https://github.com/miraheze/DiscordNotifications/commit/3d2f8d2 [00:51:14] [02DiscordNotifications] 07Universal-Omega 04deleted 031.13 at 039c22c2b 13https://github.com/miraheze/DiscordNotifications/commit/9c22c2b [00:51:18] [02DiscordNotifications] 07Universal-Omega 04deleted 031.14 at 03bcc9cef 13https://github.com/miraheze/DiscordNotifications/commit/bcc9cef [00:52:06] [02DiscordNotifications] 07Universal-Omega 04deleted 03REL1_37 at 038327878 13https://github.com/miraheze/DiscordNotifications/commit/8327878 [00:52:07] [02DiscordNotifications] 07Universal-Omega 04deleted 03REL1_38 at 038327878 13https://github.com/miraheze/DiscordNotifications/commit/8327878 [00:52:28] [02DiscordNotifications] 07Universal-Omega 04deleted 03REL1_40 at 034a96324 13https://github.com/miraheze/DiscordNotifications/commit/4a96324 [00:59:41] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/62e0ff41667f03fa865c786085e1b8e3e0f9b9a8 [00:59:42] 02ssl/03master 07MacFan4000 0362e0ff4 add kaiserreich.wiki [02:32:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:46:09] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --old=starbasewiki --new=starbasiswiki --rename (END - exit=256) [02:46:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:46:26] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --old=starbasewiki --new=starbasiswiki --rename --user BlankEclair_(Miraheze) (END - exit=512) [02:46:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:41] Starbasis? [02:48:30] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --old=starbasewiki --new=starbasiswiki --rename --user 'BlankEclair_(Miraheze)' (END - exit=0) [02:48:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:47] > Rename request for: to . [02:48:51] starbasis it seems yeah [02:48:55] okay gtg fr now [02:54:22] PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kodiak.wiki' expires in 14 day(s) (Wed 09 Apr 2025 02:48:10 AM GMT +0000). [02:54:34] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/75f4d6022a448631f99c5466a2c8936f11c989c6 [02:54:34] 02ssl/03master 07WikiTideSSLBot 0375f4d60 Bot: Update SSL cert for kodiak.wiki [03:21:14] !log [blankeclair@mwtask161] runRefreshLinks: [1300/3227] Handling kaisersvictorywiki [03:21:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:32] RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Mon 23 Jun 2025 01:55:58 AM GMT +0000. [04:02:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:42:30] PROBLEM - wiki.90percentstudios.com - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:46:53] PROBLEM - pawiki.90percentstudios.com - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:54:14] PROBLEM - ckcwiki.90percentstudios.com - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [05:22:04] !log [blankeclair@mwtask161] runRefreshLinks: [1400/3227] Handling lartdudoutewiki [05:22:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:43] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --old=shiwuyuwiki --new=thetaleoffoodwiki --rename --user 'BlankEclair_(Miraheze)' (END - exit=0) [05:51:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:14:06] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=scpclassifiedaddonwiki --new=scpclassifiedwiki --user=Reception123 (END - exit=0) [07:14:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:10] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.69, 8.82, 4.42 [07:20:05] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.72, 6.26, 3.99 [08:54:28] !log [blankeclair@mwtask161] runRefreshLinks: [1500/3227] Handling luoddawiki [08:54:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:17:34] Speedy [09:17:41] Let me get that laptop [09:17:48] And see what the rest are up to [09:19:50] @blankeclair mwtask181 crashed or done? [09:19:59] 161 is on m [09:20:08] 151 is on c [09:20:11] That ain't good [09:20:17] huh, interestingly not running anymore on 181 [09:20:19] lemme check sal [09:20:37] 171 has two running, d and e [09:20:38] > 16:03 MirahezeLSBot: [rhinos@mwtask181] runRefreshLinks: [224/2985] cravesagawiki exited with 137 [09:20:45] there you go [09:20:50] i can only act on things i get pinged on ^^; [09:21:04] Can you fix [09:21:05] though perhaps we should build a thingy to tell when things are running :p [09:21:12] Cause I'm supposed to be working [09:21:14] ah oki [09:21:23] I have salt for that [09:21:38] i can technically abuse deploykey for that :p [09:21:46] [rhinos@puppet181:~]$ sudo salt-ssh -E 'mwtask*' cmd.run 'ps aux | grep '[r]efreshLinks' [09:22:03] i have no idea what you ran runRefreshLinks on [09:22:07] your home directory is empty [09:22:13] BlankEclair: look in yours [09:22:26] it'll be the latest one in that [09:22:49] eh, screw it, it'll effectively be the same anyway lol [09:23:46] BlankEclair: /home/blankeclair/pre-1.43-shared-mwtask181-try-6.txt [09:23:52] i just made 6 [09:23:55] then 5 [09:24:02] when i've been restarting them [09:24:08] ah oki [09:24:09] !log [blankeclair@mwtask181] runRefreshLinksChunked: [0/2761] Handling cravesagawiki [09:24:11] i've been dumping the list in your home dir for you [09:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:19] oh ty ^^; [09:24:28] i have magic powers like that [09:24:37] definitely not sudo -sHu blankeclair [09:24:51] BlankEclair: not technically sure it's abusing deploykey too [09:25:19] i wonder if i should make a thingy that would gather the statistics of runRefreshLinks hourly tbh [09:25:25] not log in sal, just sends messages here [09:25:33] that key was explicity designed to allow you to be able to interact as www-data on any appserver from the active deployment server [09:25:43] oh neat [09:27:57] BlankEclair: if you can do anything outside of the scope of www-data, raise that [09:28:10] as in as a sec issue? [09:28:24] i'll debug the cargo issue first btw [09:28:29] then make the gather script [09:28:33] BlankEclair: ye [09:28:37] oki [09:29:11] but if you're just using it to mass do stuff as www-data from the deployment server, stick a script in puppet for that [09:29:32] lol yeah, makes sense [09:29:44] in this case, i'd be temporarily checking on the status of a process on four servers [09:30:02] i wonder if we can configure salt [09:32:28] BlankEclair: https://github.com/miraheze/puppet/blob/fab5b1eada83c7018a8e6b55e1978cf95cd54169/modules/salt/templates/roster.erb [09:32:28] [GitHub] [miraheze/puppet] modules/salt/templates/roster.erb @ fab5b1eada83c7018a8e6b55e1978cf95cd54169 [09:32:31] don't see why not [09:32:53] > sudo: True # Whether to sudo to root, not enabled by default [09:33:00] yeah that's not enabled by default [09:33:47] BlankEclair: ye i think you could setup salt-ssh to be able to use the deploykey [09:33:49] for mw [09:33:57] oh neat [09:33:58] i would be quite happy to support that [09:34:13] i technically could build that rn lol [09:34:27] i see no reason why not [09:34:41] there's genuine benefit [09:34:51] do you know the causes of RepoGroup::fileFile() returning false off the top of your head? [09:34:56] no [09:35:02] figured [09:35:12] it might be a miraheze thing since it uses swift, lemme test that function out to confirm [09:35:22] s/it uses swift/we use swift/ [09:35:22] BlankEclair meant to say: it might be a miraheze thing since we use swift, lemme test that function out to confirm [09:35:25] oh neat ,ty [09:37:02] BlankEclair: i don't know much off the top of my head [09:37:13] I just like to know where to find stuff / who to payer [09:37:17] yep, that function returns false for a non-existent file on test151 [09:37:22] s/payer/pester [09:37:22] RhinosF1 meant to say: I just like to know where to find stuff / who to pester [09:37:26] heh [09:37:37] BlankEclair: that's a very good skill [09:37:39] To have [09:37:43] Knowing who to annoy [09:37:44] definitely lol [09:37:49] i should go meow at lcawte [09:38:17] I don't like people much [09:38:22] But like I will say hi [09:38:26] lcawte: meowed [09:38:27] Interact at work [09:38:35] yeah, makes sense ^^; [09:38:48] I just know when to hide in my windowless computer lab too [09:39:00] when? [09:39:05] And when my blanket will be useful when I get home [09:39:17] BlankEclair: when I've dealt with too many people [09:39:21] ah [09:39:58] My computer lab is out of the way so unless you know where it is, you ain't finding me [09:40:20] secret base [09:40:30] suddenly reminded of my second minecraft world lol [09:40:34] (first one lost to the sands of time) [09:40:40] i had secret bases like everywhere [09:40:52] It's a lovely option being able to have an escape from humans [09:45:58] I should get a shower soon [09:48:21] yaron is confusing [09:48:53] he returns a `null` in the error condition of a function whose return value is always treated as a string [09:50:20] huh, it seems to work oddly enough [09:51:12] What on earth. [09:51:56] huh, it seems to work [09:51:57] wait [09:51:59] you were on discord?? [09:52:11] oops, that was an unintentional ping [09:53:55] [1/2] well, i meowed at you ^^; [09:53:55] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1354030527752966236/2025-03-25_20-53.png?ex=67e3ceb2&is=67e27d32&hm=43460348a54b69d6a9f9234f1806495dfa93121160a3864aad147e29e531bbee& [10:46:15] Censored subdomain? [10:56:10] rather not give publicity to a transphobic wiki :p [11:10:16] true [11:24:08] RECOVERY - evicnotice.peakprecision.wiki - Cloudflare on sslhost is OK: OK - Certificate 'evicnotice.peakprecision.wiki' will expire on Sun 22 Jun 2025 10:15:04 PM GMT +0000. [11:25:08] RECOVERY - outlaster.peakprecision.wiki - Cloudflare on sslhost is OK: OK - Certificate 'outlaster.peakprecision.wiki' will expire on Sat 21 Jun 2025 07:58:33 PM GMT +0000. [11:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:22:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:46:14] !log [blankeclair@mwtask161] runRefreshLinks: [1600/3227] Handling metaversewiki [12:46:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:46:27] there's a fucking what wiki [13:14:03] !log [rhinos@mwtask171] runRefreshLinks: [126/2589] esnarutowiki exited with 137 [13:14:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:20:38] guess i'll handle that [13:22:53] !log [blankeclair@mwtask171] runRefreshLinksChunked: [0/2463] Handling esnarutowiki [13:22:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:32] imma go eep now, nini~ [13:33:14] Ty BlankEclair and night [13:33:20] I should stalk rhinos [13:41:15] What [13:41:59] So I get pinged when why shell scripts exit [14:47:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:27:17] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49931MiB (10% inode=99%); [15:58:35] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/3ab8b71daea6c54c743c82ae6b6d6fab741f535e [15:58:36] 02ssl/03master 07MacFan4000 033ab8b71 add wiki.monsterbrothel.com [18:47:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:56:52] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/b57f7525c36b3c686a401fc31af766d998af13ef [18:56:52] 02ssl/03master 07MacFan4000 03b57f752 add serin.wiki [19:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:40:31] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/5c8574cb9432ae09ac0f593af05297749f3572bf [19:40:31] 02ssl/03master 07MacFan4000 035c8574c add wiki.starbasis.com [21:43:50] !log [blankeclair@mwtask171] runRefreshLinksChunked: [100/2463] Handling findtheplugswiki [21:43:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:00] ig it's a good time to build that central stalking thing [22:49:17] i go to fucking sleep and discord has a new ui [22:57:02] how tf does discord make devtools _lag_ [23:06:27] uoooooh discord,,, https://files.sharkgirl.ing/~claire/2025-03-26_10-04.png [23:34:10] oh yes, I love when it does that whenever I opened dev tools in the past lol [23:34:27] always wondered if they do this on purpose so you don't open dev tools