[15:36:47] <Sarrus>	 I get an "internal error" when trying to edit using source editing in da.wikipedia but not with visual editor. When opening the source edit in sections there's no error. In personal sandbox there's no error, but when trying user page the problem is there
[15:38:41] <RhinosF1>	 Sarrus: do you have a session number? (Set of letters and numbers shown at the start of the error in square brackets)
[15:39:12] <Sarrus>	 [ec5527e2-100f-42e7-8c97-297b34f572ac] 2023-05-31 15:38:59: Fatal undtagelse af typen "Exception"
[15:39:18] <Sarrus>	 in Danish
[15:39:43] <RhinosF1>	 Yes that
[15:39:49] <Sarrus>	 ok
[15:40:17] <RhinosF1>	 Sarrus: does the error persist or was it a one off?
[15:40:31] <RhinosF1>	 The session id will change but that doesn't matter
[15:41:21] <Sarrus>	 I mentioned it 15.42 (CEST) in #wikipedia-da
[15:41:41] <RhinosF1>	 I'm not in there
[15:41:54] <RhinosF1>	 If it's persistent, when did you first notice it?
[15:41:54] <Sarrus>	 but has not checked it in the time betweem
[15:41:57] <Sarrus>	 between
[15:42:15] <RhinosF1>	 Can you check and see if it is still happening?
[15:42:20] <Sarrus>	 it was just to say that I observed inmate that time
[15:42:28] <Sarrus>	 *observed it in
[15:42:41] <RhinosF1>	 Just trying to work out how quickly it should be looked at though
[15:42:47] <Sarrus>	 yes, it's still there
[15:43:03] <RhinosF1>	 And when did you first notice it?
[15:43:30] <Sarrus>	 about the time I wrote it in wikipedia-da
[15:43:45] <Sarrus>	 Kaare also has the problem 
[15:43:56] <RhinosF1>	 Interesting
[15:44:02] <RhinosF1>	 There's been no train yet today
[15:44:11] <Sarrus>	 train?
[15:44:36] <RhinosF1>	 Weekly deployment cycle
[15:44:42] <Sarrus>	 ah..
[15:44:49] <RhinosF1>	 dawiki gets its updates on a Wednesday
[15:46:51] <RhinosF1>	 I will try and find someone to help but a phab task is probably best
[15:47:59] <Sarrus>	 I have never used Phabricator
[15:48:37] <Sarrus>	 I have mentioned it in https://da.wikipedia.org/wiki/Wikipedia:Teknisk_forum#Intern_fejl to hear if others have the problem or not
[15:50:08] <RhinosF1>	 I can help get one filed
[15:50:33] <RhinosF1>	 I'm seeing if can gather some info first
[16:00:38] <TheresNoTime>	 Sarrus: did you have this issue on https://da.wikipedia.org/wiki/Stuart_O%27Grady ?
[16:19:32] <Kaare>	 TheresNoTime: it's happening anywhere it seems; hit edit on an article, and you see the exception - I noticed that it says "Fra {{SITENAME}}, den frie encyklopædi"
[16:20:31] <RhinosF1>	 Kaare: do we know if this is new today?
[16:20:51] <Kaare>	 Well, it didn't happen yesterday evening when I was last online
[16:21:01] <Kaare>	 (that's in CEST)
[16:21:23] <RhinosF1>	 TheresNoTime: something wider is wrong here
[16:21:32] <RhinosF1>	 I'm inclined to make this UBN
[16:21:40] <RhinosF1>	 5/5 pages I just tested failed
[16:22:04] <TheresNoTime>	 Really? I just went to https://da.wikipedia.org/wiki/Efficiens and clicked edit, it's fine
[16:22:05] <Sarrus>	 TheresNoTime: Yes
[16:22:19] <Sarrus>	 the problem is also on that page
[16:22:21] <RhinosF1>	 TheresNoTime: not for me
[16:22:34] <RhinosF1>	 TheresNoTime: [3479279c-d9a0-4ebf-a79d-e04403a74ef6] 2023-05-31 16:22:11: Fatal exception of type "Exception"
[16:22:41] <TheresNoTime>	 Logged in?
[16:22:52] <Sarrus>	 TheresNoTime: I also have the problem on that page
[16:23:14] <Sarrus>	 could it be different if I log out?
[16:23:15] <RhinosF1>	 TheresNoTime: yes
[16:23:42] <Kaare>	 a user said they get the error when they attempt to log in
[16:24:04] <Sarrus>	 Kaare: where?
[16:24:10] <RhinosF1>	 TheresNoTime: [607c681a-8fea-4490-b9ac-c8394dc1b530] 2023-05-31 16:23:52: Fatal exception of type "Exception" logged in
[16:24:18] <TheresNoTime>	 RhinosF1: https://da.wikipedia.org/wiki/Efficiens?action=edit&uselang=en does that work?
[16:24:32] <Kaare>	 Sarrus: Diskussion:Lockheed Martin F-35 Lightning II
[16:24:32] <RhinosF1>	 TheresNoTime: yes
[16:24:52] <Kaare>	 I also get the exception when not logged in (different browser)
[16:24:53] <Sarrus>	 TheresNoTime: then there's no error
[16:25:46] <RhinosF1>	 TheresNoTime: could it be something with da wrong
[16:25:48] <Kaare>	 oh, so it's a broken translation?
[16:26:29] <RhinosF1>	 TheresNoTime: something is seriously breaking dansk
[16:26:32] <Sarrus>	 should it be fixed on translatewiki then?
[16:26:41] <RhinosF1>	 Sarrus: unlikely a translation
[16:26:47] <Sarrus>	 ok
[16:27:41] <RhinosF1>	 Sarrus: do you know of another language using the same encoding
[16:28:12] <Kaare>	 Norwegian has the same letters as us, if that's what you mean
[16:28:27] <Sarrus>	 don't know what an encoding is
[16:28:32] <RhinosF1>	 Kaare: yes
[16:28:39] <Sarrus>	 ah
[16:29:10] <TheresNoTime>	 put T337700 as UBN
[16:29:11] <stashbot>	 T337700: Exception: preg_match_all error 4: Malformed UTF-8 characters, possibly incorrectly encoded - https://phabricator.wikimedia.org/T337700
[16:29:19] <RhinosF1>	 Hmm
[16:29:25] <RhinosF1>	 I can't reproduce on nowiki
[16:29:32] <Sarrus>	 I have no problems in nowiki
[16:30:52] <Sarrus>	 or nowiktionary
[16:32:02] <RhinosF1>	 Im causing a fuss but for now change your language to English
[16:33:39] <Sarrus>	 can't open the settings page either
[16:33:50] <Sarrus>	 https://da.wikipedia.org/wiki/Speciel:Indstillinger
[16:34:03] <RhinosF1>	 https://da.wikipedia.org/wiki/Speciel:Indstillinger?uselang=en
[16:34:03] <sukhe>	 RhinosF1: thanks for the heads up
[16:34:06] <RhinosF1>	 Sarrus: ^
[16:34:22] <Sarrus>	 super :-)
[16:34:22] <RhinosF1>	 sukhe: seems to also break special pages too.
[16:34:34] <RhinosF1>	 I've alerted releng
[16:34:56] <Kaare>	 I get the exception immediately after a completed login
[16:35:15] <RhinosF1>	 Kaare: yeah the bug seems to be major
[16:35:22] <RhinosF1>	 I'm not sure why it would happen today
[16:35:54] <Kaare>	 yup, and it seems that I wasn't actually logged in
[16:35:57] <RhinosF1>	 sukhe: it can't be the train if it started today so is there anything SRE wise that could impact
[16:35:59] <Sarrus>	 grr, I get the error when clicking the save bottom in settings
[16:36:20] <sukhe>	 RhinosF1: I don't seen in anything the changes
[16:36:33] <RhinosF1>	 It's stopping users even login on at least 1 language so we're kinda a fairly major impact
[16:37:10] <Sarrus>	 so I can't change the interface language
[16:38:06] <Kaare>	 Sarrus: confirmed
[16:38:13] <Sarrus>	 now it changed
[16:38:30] <Sarrus>	 Kaare: what's confirmed?
[16:38:48] <Sarrus>	 but it makes it possible to edit
[16:39:09] <Kaare>	 can't save
[16:39:24] <RhinosF1>	 Sarrus: have you been able to get it to save?
[16:39:38] <Sarrus>	 suddently it changed til English
[16:39:47] <RhinosF1>	 Ah
[16:39:49] <Sarrus>	 *to
[16:40:03] <Sarrus>	 could it be a question of skin?
[16:40:16] <RhinosF1>	 I'm not sure
[16:40:24] <RhinosF1>	 This is beyond my technical knowledge
[16:41:16] <Sarrus>	 <joke>even the articles changes to English: https://da.wikipedia.org/wiki/Deux_mille</joke>
[16:41:22] <Sarrus>	 :-)
[16:41:42] <Sarrus>	 deleted now
[16:42:53] <Kaare>	 my notification counter isn't parsed at first: "{{PLURAL:1|Notice (1)|Notices (1)|100=Notices (99+)}}", but once clicked it changes to the expected "1" - same error, I'm sure
[16:43:21] <RhinosF1>	 Hmm
[16:43:25] <RhinosF1>	 Something is very broken
[16:53:51] <TheresNoTime>	 It's a message, the `templates used on this page` message
[16:54:41] <RhinosF1>	 TheresNoTime: how
[16:54:45] <RhinosF1>	 What
[16:55:05] <Sarrus>	 https://translatewiki.net/wiki/MediaWiki:Templatesused/da ?
[16:55:20] <TheresNoTime>	 https://da.wikipedia.org/wiki/MediaWiki:Templatesused
[16:55:41] <Sarrus>	 what to do?
[16:56:21] <Kaare>	 that's an old message
[16:56:40] <Sarrus>	 should it be deleted?
[16:58:01] <TheresNoTime>	 We can try? I just tried to force through an edit and it "saved", but didn't save
[16:58:11] <RhinosF1>	 TheresNoTime: CLI
[16:58:59] <TheresNoTime>	 deleted, try editing now?
[16:59:14] <Kaare>	 Seems good now
[16:59:42] <RhinosF1>	 TheresNoTime: settings page still broken
[16:59:58] <TheresNoTime>	 RhinosF1: which?
[16:59:58] <RhinosF1>	 Try https://da.wikipedia.org/wiki/Speciel:Indstillinger?uselang=da
[17:00:04] <RhinosF1>	 TheresNoTime: preferences
[17:00:20] <RhinosF1>	 Kaare, Sarrus: can you reproduce?
[17:00:24] <TheresNoTime>	 ack, looking
[17:00:35] <Kaare>	 yes, preferences still broken
[17:00:55] <RhinosF1>	 TheresNoTime: 133b5197-9dda-48ac-ba51-6564efa79dff] 2023-05-31 16:59:21: Fatal exception of type "Exception"
[17:02:23] <Kaare>	 so, there's likely more such old messages with broken encoding?
[17:03:12] <RhinosF1>	 Potentially ye
[17:03:50] <TheresNoTime>	 https://da.wikipedia.org/wiki/MediaWiki:Tog-previewonfirst
[17:04:39] <RhinosF1>	 TheresNoTime: how are you finding?
[17:04:44] <RhinosF1>	 But please delete
[17:05:08] <Kaare>	 oddly, the edit summaries are not broken
[17:06:03] <TheresNoTime>	 why have these broken :/ https://da.wikipedia.org/wiki/MediaWiki:Tog-previewontop
[17:06:34] <TheresNoTime>	 RhinosF1: did a verbosely logged request to that page, searched `channel:parser` in logstash
[17:06:54] <RhinosF1>	 TheresNoTime: ah
[17:07:19] <RhinosF1>	 It doesn't make sense why they've suddenly broken
[17:07:29] <TheresNoTime>	 think prefs is working again now, but I won't believe its just da.wiki which is going to suffer from this..
[17:08:06] <Kaare>	 yes, the preferences are displayed again
[17:08:11] <RhinosF1>	 TheresNoTime: no, I'd like some deeper investigation
[17:08:59] <TheresNoTime>	 primarily hu.wiki and da.wiki it seems..
[17:09:11] <RhinosF1>	 I'm inclined to stay at UBN until too
[17:09:19] <RhinosF1>	 Because it simply doesn't make sense
[17:09:33] <TheresNoTime>	 Kaare: what is da.wiki's "administrator noticeboard" or similar? I should make a note there that I deleted pages
[17:09:53] <RhinosF1>	 And Sysadmin randomly being bribed to delete pages isn't going to work if it's a wider scope
[17:11:17] <Kaare>	 TheresNoTime: I'm not actually sure what we use for that; instead perhaps a follow-up in https://da.wikipedia.org/wiki/Wikipedia:Teknisk_forum#Intern_fejl ?
[17:11:28] <TheresNoTime>	 sure okay :)
[17:11:38] <RhinosF1>	 TheresNoTime: can you link in task too
[17:11:43] <RhinosF1>	 And probably log in SAL
[17:12:13] <Sarrus>	 https://da.wikipedia.org/wiki/Wikipedia:Anmodning_om_administratorassistance ?
[17:12:57] <Kaare>	 well, that's more the other way 'round (asking for admin assistance)
[17:13:19] <Sarrus>	 isn't it what is requested?
[17:13:50] <Sarrus>	 ah hadn't read it all
[17:14:02] <RhinosF1>	 They've been deleted
[17:14:14] <RhinosF1>	 Sysadmins just very rarely take Sysadmin actions
[17:14:18] <RhinosF1>	 So it's normally explained
[17:14:29] <Reedy>	 We're supposed to explain when we do stuff onwiki? :P
[17:15:14] <Sarrus>	 well, then it probably should the the page Kaare linked to
[17:15:31] <Sarrus>	 that's the technical forum
[17:16:09] <Sarrus>	 but we also have a general notice board at https://da.wikipedia.org/wiki/Hjælp:Opslagstavle
[17:17:13] <TheresNoTime>	 hopefully https://da.wikipedia.org/wiki/Special:Diff/11489388 is enough :)
[17:17:14] <Sarrus>	 the problem is solved
[17:18:12] <Kaare>	 thanks, TheresNoTime - Sarrus and I can explain further if need be :-)
[17:18:13] <RhinosF1>	 Reedy: do you know anyone who might understand the bug?
[17:18:19] <TheresNoTime>	 :)
[17:18:42] <TheresNoTime>	 it is still heavily affecting hu.wiki
[17:19:44] <Sarrus>	 I can't find the diff-number any more for the Special:Diff - it just say "prev" or "next" - where do you find it, TheresNoTime?
[17:21:20] <Kaare>	 TheresNoTime: yikes, is it the same messages? - I was wonderering whether the message had been unused for years but recently revived
[17:22:23] <TheresNoTime>	 Sarrus: for Special:Diff/11489388 ? I use https://en.wikipedia.org/wiki/User:Enterprisey/diff-permalink in my `global.js` so it adds a nice easy diff number to copy
[17:22:49] <TheresNoTime>	 Kaare: I've not looked (it takes a bit of effort to find each of the failing messages..), but it could be..
[17:25:38] <Sarrus>	 TheresNoTime: it calls a template on enwiki. Does that work on meta?
[17:26:06] <Sarrus>	 {{subst:iusc|User:Enterprisey/diff-permalink.js}}
[17:26:33] <TheresNoTime>	 I just have `mw.loader.load('//en.wikipedia.org/w/index.php?title=User:Enterprisey/diff-permalink.js&action=raw&ctype=text/javascript');` in my meta `global.js` for it :)
[17:26:44] <Sarrus>	 ah
[17:31:12] <Sarrus>	 it works :-)
[17:39:43] <Kaare>	 TheresNoTime: I've added an explanation to the page in Danish; you're more than welcome to delete any more defective messages you might come across
[17:40:59] <TheresNoTime>	 Thank you :)
[17:42:54] <Kaare>	 TheresNoTime: I just noticed that logins are still failing
[17:43:14] <Kaare>	 [6df024af-8166-4df1-810b-0756f5c15dc9] 2023-05-31 17:42:21: Fatal undtagelse af typen "Exception"
[17:44:08] <TheresNoTime>	 Kaare: ack, will look..
[17:45:30] <Kaare>	 my notification counter is also still broken (don't have a UUID there)
[17:48:07] <TheresNoTime>	 yup it was https://da.wikipedia.org/wiki/MediaWiki:Loginsuccess (logins should work now?)
[17:48:42] <Kaare>	 just tried it, and it works - thanks
[17:50:47] <Kaare>	 right now my notification counter looks okay, not sure what I did before to get the brokenness
[17:58:30] <Kaare>	 TheresNoTime: don't know if the message is used, but I randomly found and deleted MediaWiki:Loginreqtext containing the same error
[18:02:38] <TheresNoTime>	 Kaare: ah thank you..
[18:09:17] <TheresNoTime>	 Kaare: can you fix https://da.wikipedia.org/w/index.php?title=Kategori:Violette_blomster&action=edit ?
[18:12:00] <Kaare>	 done
[18:12:13] <Kaare>	 or not
[18:13:18] <Kaare>	 now done using English
[18:13:40] <TheresNoTime>	 ah no it won't save will it..
[18:13:54] <Kaare>	 switching to English fixed it
[18:15:15] <Kaare>	 it's easy to find more defective messsages; I navigated back in time in the message creator's contributions...
[18:18:12] <RhinosF1>	 tgr_: discussion here
[18:18:19] <RhinosF1>	 But no, we have no idea why now
[18:21:29] <Kaare>	 I came across a redirect page with the broken encoding, but the redirect actually works: https://da.wikipedia.org/w/index.php?title=Oversaettelse
[18:27:30] <TheresNoTime>	 I'm not sure if https://phabricator.wikimedia.org/T337700#8893364 is the best approach, but maybe malformed characters is better than exceptions at this point?
[18:32:02] <Kaare>	 malformed characters would be better in my opinion
[18:33:21] <Kaare>	 I suspect all of the dawiki user's contributions from that period of time will appear broken
[18:36:43] <RhinosF1>	 Amir1: are you sure it's not related to legacy encoding? See https://phabricator.wikimedia.org/T337700#8893404
[19:11:02] <RhinosF1>	 Amir1: thanks for reply and yes any help is useful
[19:11:17] <RhinosF1>	 After you've fought any fires
[19:11:56] <Amir1>	 yeah, dealing with T337734 rn
[19:11:57] <stashbot>	 T337734: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734
[20:39:07] <KnudW>	 TheresNoTime - Re the situation on dawiki: It looks like we've lost the posibility to patrol new pages in all namespaces. The link's gone and there's a "[]" instead.
[20:39:07] <KnudW>	 Could this be related to the current issue and  the blanking/deletion of the old pages?
[20:41:47] <TheresNoTime>	 KnudW: could you link to an example?
[20:42:26] <KnudW>	 https://da.wikipedia.org/wiki/Brugerdiskussion:152.115.160.202
[20:47:17] * TheresNoTime looking
[20:51:28] <TheresNoTime>	 KnudW: yup, can replicate (and by doing `?uselang=en` the message shows)
[20:52:40] <KnudW>	 Aha.
[20:53:29] <TheresNoTime>	 KnudW: try now?
[20:53:42] <TheresNoTime>	 (deleted https://da.wikipedia.org/wiki/MediaWiki:Markaspatrolledtext)
[20:54:38] <KnudW>	 Yup, it's back and I've just patrolled that page
[20:55:31] <TheresNoTime>	 good :)
[20:55:33] <KnudW>	 Thanks, I'll pass it on. 
[20:56:36] <TheresNoTime>	 KnudW: while you're here, https://da.wikipedia.org/wiki/Battle_of_the_Beam
[20:56:44] <TheresNoTime>	 probably needs deleting :p
[20:58:17] <KnudW>	 Aha, https://da.wikipedia.org/wiki/Speciel:Bidrag/80.198.203.137?uselang=da.
[20:58:25] <KnudW>	 https://da.wikipedia.org/wiki/Speciel:Bidrag/80.198.203.137?uselang=en
[20:59:21] <Izno>	 lol, that one in timeless in ?uselang=da is fun
[20:59:26] <TheresNoTime>	 ack, looking..
[21:02:46] <TheresNoTime>	 KnudW: resolved? https://da.wikipedia.org/wiki/MediaWiki:Nocontribs
[21:03:17] <KnudW>	 Yes.
[21:04:04] <TheresNoTime>	 I have no idea why this has happened :((
[21:06:36] <KnudW>	 I'm thinking cosmic rays until someone comes up with something better...