[01:13:16] tgr: Thanks for . I had never read that before. It hits a bit too close to home, but it does ring true. [02:33:47] Maybe other folks would like to chime in: https://meta.wikimedia.org/wiki/Grants_talk:Project/Isarra/Post-deployment_support#Round_2_2017_decision [08:50:20] no_justification: You know your username tries to undermine the point you made, don't you? :P [08:53:41] I haven't found the right signature for my new identity yet [08:54:11] Ch😂d? [09:01:31] I like it! [15:35:50] addshore: DanielK_WMDE_: (Moving here because both -tech and -dev are noisy) During my walk I figured out WHY the "revision does not exist" message happened. It was related to ChronologyProtector, but not in the way we thought: it happened BECAUSE it was doing its job [15:36:09] :D [15:36:19] For newly created pages/revisions, the text table rows were written to the replica (and so weren't on the master), but the revision table rows were written to the master (and replicated to the replica) [15:36:50] When you first save the page, the replica hasn't caught up yet, so CP ensures that your next page view reads from the master (which never happens otherwise), because it's the only up-to-date server [15:37:08] The master doesn't have the text row, so trying to get the text fails [15:37:27] Then when you refresh, the replica has caught up, so your page view reads from the replica, and it has both rows so it works fine [15:37:34] aaah, and then the second refresh reads from the replica [15:37:49] Sounds like a thoughtful walk :) [15:37:51] The real problems began when an AbuseFilter rule was hit, because AF was still writing to the master [15:38:23] So the master assigns that text row an old_id which it thinks is the next available old_id, but the replica has already used that ID for something else [15:38:43] and then that is the point the replication exploded [15:38:44] Then when the replica tries to replicate that insertion, it fails because of an ID collision, and replication stops [15:39:25] Leaving both the replica and the master in a broken state: the master has revision rows pointing to old_ids that don't exist, or if they do, point to AbuseFilter data [15:39:56] RoanKattouw: so wikidatawiki on beta is also broken [15:40:00] And the replica has one AbuseFilter log entry that points to an old_id that points to revision text (and no more, becaues replication stops at this point) [15:40:05] Yeah I can imagine [15:40:10] I'm just about to check the others [15:40:20] https://phabricator.wikimedia.org/T183232 [15:40:26] well that ticket actually talks about enwiki [15:40:38] I fixed enwiki and deploymentwiki by transferring the text rows from the replica to the master and updating the references to them in the revision table for their new IDs [15:40:51] you can probably write a query to find all wikis that have edits between now and the time the patch was first merged / landed on beta [15:41:43] I could do that but it's easier to just compare the text tables on the master and replica [15:41:58] If there are rows they disagree on, that means I need to fix things [15:42:11] Going to start doing that with wikidatawiki now [15:42:48] well, sorry for this fallout, and thanks for helping! [15:43:17] No worries! [15:43:29] 4 layers failed here [15:43:43] (Author, reviewer, MW DB abstraction, read only flag on the DB server) [15:43:57] So I can hardly blame any individual one [15:46:54] Jeez, the divergence on wikidata is at least 200 rows [15:47:09] RoanKattouw: browser tests [15:47:33] Ugh [15:48:13] Looks like they're all sequential though, no AbuseFilter entries in between [15:48:32] Which will make it easy to generate the query to insert the missing rows [15:49:20] :) [15:49:35] Justed filed another bug about wikidata beta RC being broken due to ORES [15:53:59] What did ORES break there? [15:55:28] OK Wikidata's revision/text should be fixed now [15:59:30] commons is also done, now working on dewiki [15:59:34] :D [16:05:31] Now I just need to do zhwiki, the others are all clean [16:06:14] Oh, nm, zhwiki isn't corrupted, it just looked like it might have been because the latest event was an AF entry, not a revision [16:06:25] Everything's clean so I'm going to disable read-only mode [16:10:15] wheeee, awesome [16:19:43] RoanKattouw: thanks a ton for all your help! [16:21:22] beta is back up now and accepting writes [16:21:39] Unfortunately it's now also on a single DB server [16:22:04] RoanKattouw, addshore: I added a log from here to the ticket: https://phabricator.wikimedia.org/P6486 [16:22:21] I thought it's useful to preserve Roan's analysis [16:22:26] DanielK_WMDE_: https://phabricator.wikimedia.org/P6485 :P [16:22:29] Yeah thanks for that [16:22:39] oh wait, you mean this latest snippet! [16:22:57] Not sure if it's quite the right task, but since it's a pastie I suppose you can embed it on multiple tasks [16:23:02] Right, im done with meetings and now back to MCR stuff [16:25:09] addshore: I swear I wrote an email about the revert a few hours ago. I can't find it any more. I'm REALLY getting annyoed with Thunderbird. [16:25:14] addshore: did you get the mail? [16:26:09] RoanKattouw: yea... I didn't see a better one [16:26:57] AFAICT nobody ever filed a task about the original breakage when saving in beta [16:27:19] yea [16:31:26] addshore: did you? [16:31:33] RoanKattouw: i dont think we did no [16:31:54] DanielK_WMDE_: and i dont remember seeing your mail, ut i saw amanda's reply to one [16:32:00] my inbox if rather full right now thouhg [16:32:07] i had an unexpected drive accross the country last night [16:32:39] amandas reply to one? [16:32:41] huh? [16:32:45] what? when? where? [16:32:48] now I'm more confused... [16:33:23] I didn't see any reply. and i don't see the mail. it just... vanished? [16:33:33] fucking technology... [16:33:35] I have one today with the title "Reverting MCR storage layer patch" [16:33:38] *subject [16:33:59] But I dont see your origional one "Tue, Dec 19, 2017 at 7:59 AM, Daniel Kinzler" [16:34:02] ah! that sounds good. can you send that to me? because I no longer have it. and i have no idea how that happened [16:34:21] DanielK_WMDE_: you didnt use your wikimedia.de email account, thats probably why [16:34:33] i expect your looking in the wrong place :) [16:35:14] i'm looking in all the places [16:36:43] DanielK_WMDE_: I just forwarded you one [16:36:43] DanielK_WMDE_: I've +2ed your DB nowrite patch and added a comment suggesting how you might implement the master wrapping thing [16:36:58] DanielK_WMDE_: Also it would be nice to have unit tests showing that wfGetDB( DB_REPLICA )->insert() does in fact throw [16:37:36] RoanKattouw: yes, but that's not the case yet if what youget from that call is the actual master connection. whcih, for unit tests, is always the case. [16:37:54] So, yea, once the wrappign thing is there, we can have such a test [16:39:00] lol good point [17:14:25] "1.30 has several database changes since 1.29, and will not work without schema updates. Note that due to changes to some very large tables like the revision table, the schema update may take a long time (minutes on a medium sized site, many hours on a large site)." <-- what is considered "large" [17:24:53] wikipedia-sized [17:26:12] ok so a wiki with 280K pages and 745K revisions isn't going to take hours [17:26:12] although the exact time taken by schema updates depends on both how large the db is and what sort of resources your db server is able to use (IOPS, CPU, RAM, etc.). [17:26:55] SantaC: I would guess minutes, not hours for that size [17:27:03] if you're on 7.2k spinning rust using 1GB of RAM (so you're swapping like mad), and you have a 1 core 1.8GHz CPU that could take hours for a site of that size :) [17:27:43] but yeah, a more realistic estimate is a few minutes [17:28:39] (another thing that can bog it down is if the webserver where you're running the update script and the db server are two different servers and the internet connection between them is best described as a silly straw) [17:29:31] we use Amazon RDS. the database is around 100GB in size and the webserver is also on AWS as an EC2 instance [18:47:14] o/ [18:47:27] Is this were the MediaWiki Platform team hangs out? [19:11:44] halfak: sounds about right :D [19:12:05] note tim's on vacation, not sure who else is working today :D [20:17:46] Woo it's official then. [20:18:12] I dunno about you but I feel incredible excitement reading that topic for this channel [20:28:44] Skizzerz: 13 minutes, more than I thought! [21:01:36] Eek! Missed brion's ping. [21:01:44] \o/ [21:02:04] So I am wondering if something is within the scope/radar of MediaWiki Platform. [21:02:06] * halfak gets link [21:02:28] I'm working on a new content contribution extension and I'm reading https://en.wikipedia.org/wiki/User:Risker/Risker's_checklist_for_content-creation_extensions [21:03:23] Any chance that there's some clean integration with all of the awesome MW curation bits for new extensions? [21:03:46] heh [21:03:51] E.g. "put everything in this place, it'll get added to RC and Logging and start showing up in all of the curation/suppression stuff" [21:03:51] that would be easy wouldn't it :) [21:03:55] lol right? [21:04:19] general advice: put things in page content, and you inherit all that stuff [21:04:24] SO secondarily, if I wanted to do this right is that something y'all would be interested in helping us with? [21:04:37] if you create your own tables or ways of storing things separately from revisions, then you may have to recreate more tools [21:04:59] brion, when you say "put it into page content" -- how flexible is that? [21:05:23] Like, can I have pages that you would never really render? [21:05:25] well you can stuff any structured data into a Content blob (and JSON or whatever it on the backend) ike say wikidata does [21:05:30] and then have custom rendering for it [21:05:44] Oh good point. [21:05:55] i'd say we're interested in making sure people have the tools to add tools to the system :D [21:06:16] Cool. That's primarily what I need now. [21:06:40] halfak: great, drop cindy cicalese a note so it's on radar [21:06:45] I'd like to bring some things to this team mid-next-quarter for advice and discussion. [21:06:50] Is she on IRC ever? [21:06:51] and if we have to shuffle we can :D [21:07:03] CindyCicaleseWMF: poke [21:07:15] Haha. Most obv username ever ^_^ [21:07:19] :D [21:07:25] if she's offline, drop her an email [21:07:34] Will do. Thanks brion :D [21:07:49] anyway we won't write your extension for ya, but if we can help make sure there's a defined way of doing things, that's great [21:08:05] Oh yeah totally. [21:08:10] :) [21:08:40] TOOLS for TOOLS [21:09:14] tools to the toolsth power [21:15:26] wow [21:15:31] WMF in an IRC nickname [21:22:15] Thanks for the poke, brion :-) I'm here! Reading the scrollback now . . . [21:25:48] Yes, most obvious username ever ;-) I'm not an IRC native, so I'm still learning the IRC way . . . [21:27:56] halfak, I'm happy to chat about your ideas whenever you'd like [22:54:42] o/ CindyCicaleseWMF [22:54:46] Sorry I missed the ping. [22:54:59] Just happened to be clicking through my channels and saw your message. [22:55:30] Generally, right now I'm thinking really hard about how to cram our contribution pattern into page/revision form. [22:56:01] It doesn't make sense for our schema, but from our point of view, MW is just a client. So in the end, I don't think it needs to make perfect sense. [23:09:15] halfak: https://www.mediawiki.org/wiki/Everything_is_a_wiki_page#Examples EventLogging is usually the canonical example for putting JSON into a wiki page