[01:35:34] Hi ^demon. [01:35:59] ^demon: Any movement on getting reviewers counts from Gerrit? [01:36:19] <^demon> I got completely sidetracked today trying to figure out why gitblit is all explodey :( [01:36:28] :-( [01:37:09] <^demon> I'm totally cool with doing it though, and we will :) [01:37:36] I'd certainly like to avoid the hits to the API. :-) [01:38:45] <^demon> Absolutely. [03:11:38] Apparently 4-5 hours ago, wm-bot2 disconnected somehow (netsplit is my guess) and @restart isn't restarting it.. [03:12:26] Can someone restart it manually? [03:12:31] Thanks... [03:14:14] petan, ^ [03:20:43] Krenair: I would think Petr would be asleep this time of day... [03:20:57] That is why I said anyone... [03:21:09] Do you have access to do it? [03:21:42] I know where directions are if you have access. [03:22:04] Nope. It's on the bots project. I don't have access there [03:22:21] link to the instructions here though, maybe someone else does [03:22:21] ... [03:22:40] Just a sec... let me pull up my bookmark. [03:23:35] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot [03:24:05] !reboot [03:24:13] !reboot is https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot [03:24:14] Key was added [03:25:16] There.. easy to get if wm-bot isn't the one that is split... [03:27:10] wm-bot: says it is shutting down, but doesn't do it. [03:49:45] I have a technical problem. [03:50:21] I *should* be global IP block exempt on all WMF projects, but I'm blocked on en.wp. [03:50:24] Can anyone help? [09:58:01] hey [09:58:20] ip from wikimedia range is shown on anon edits [09:58:21] https://en.wikipedia.org/wiki/Special:Contributions/208.80.154.75 [09:59:36] there was a similiar case https://bugzilla.wikimedia.org/show_bug.cgi?id=48919 [10:12:40] you will get more takers in about [10:12:43] * apergos looks at the clock [10:12:50] 7 hours I guess [10:13:03] when sf gets on line [10:37:58] hmpf. Is the sul problem fixed? [10:42:26] this wording makes it sound like it's sul fault and it'd be good to fix by removing sul :) [10:42:31] i like the irony of that somewhat [10:44:21] apergos: you may want to investigate the issue or raise it with others instead of waiting 8 hours? [10:44:38] I am looking but I don't remember how it works [10:44:58] so I have to look up everything from the beginning again [10:45:53] we're all up and active aren't we? [10:46:33] yes but this was a mw core issue last time [10:46:45] not something ops people solved iirc [10:47:19] this was simply an issue of new servers not being in the XFF list [10:47:24] that's the first thing to check [10:47:41] and last time I checked, there are platform people in europe too [10:50:42] it is not possible to work... evry hour i am logged out... [10:51:02] Logged in on Commons, Logged out on Meta, Wikipedia etc.... grrr [10:54:56] I'm sorry but looking at the bug did not jog my memory as to that list's existence and/or what it does, so it was not even on my radar of things to check [10:55:04] this wording makes it sound like it's sul fault and it'd be good to fix by removing sul :) >.> [12:04:01] any change done to the commons api? [12:10:04] matanya: can you be more specific ? [12:10:26] there are changes to the software all the time. it's big.... [13:18:49] thedj: pikiwiki project can't push submmitted images to commons from the recent days [13:23:43] Steinsplitter: hear hear, I'm having the same problem [13:24:10] Logged in on one project, loged off on another, then logged out after X time [13:29:11] twkozlowski: a lot of users hav the same problem [15:07:35] chrismcmahon: are you around? [15:07:48] hi matanya [15:09:05] hi chrismcmahon. The pikiwiki project has reported the commons API changed in the last few days and they can no longer upload files to commons. they would like to have a technical call with an engneer to sort this out. [15:09:11] Is this possible? [15:10:10] matanya: a call is probably not possible. I would suggest you file a ticket in Bugzilla [15:11:12] marktraceur: ^^ does that sound familiar? [15:11:34] Maybe. [15:11:53] YuviPanda did run through and leave the broken corpse of campaigns behind [15:12:03] Let me check it out [15:12:37] matanya: Where are they trying to upload, via a 3rd-party client or through UW? [15:12:55] thanks chrismcmahon, though to be honest i'm not sure where or why the problem lays [15:13:01] marktraceur: 3rd [15:13:23] http://www.pikiwiki.org.il/ --> marktraceur [15:14:03] matanya: Is there source code for the client somewhere? [15:14:34] not that i know, but i think i can get the dev in conntact [15:14:41] Since I cannot read or even recognize that language [15:14:46] hebrew [15:14:53] *nod* now I know, ta :) [15:15:30] marktraceur: http://www.pikiwiki.org.il/index.php?action=content&id=21 [15:15:37] see the about in english [15:16:31] K, cool [15:16:48] matanya: Yeah, I'd love to see the source code for the uploader [15:17:25] i'll try to get it for you and upload it to some public place or send you by mail. [15:17:35] K, thanks [15:17:52] mtraceur {at} member {dot} fsf {dot} org [15:20:18] cool, great [15:48:19] andre__: greg-g - heads-up re the possible API breakage reported earlier in this channel by matanya [15:49:52] needs more specific info [15:51:08] heh [15:51:38] * Technical_13 needs a nap. JavaScript bores the crap outta me... [15:51:52] andre__: how can i help? [15:52:02] and thanks sumanah [15:53:00] defining what "Commons API changed" :) [15:53:39] hmm, i'm not sure. i'll try to find out with their dev [15:54:50] andre__: "something changed", "something reverted" [15:55:46] Reedy: ah, "something"! Gotcha! ;) [15:55:51] :) [15:56:45] matanya: thank you for reporting the problem [15:58:21] matanya, bug report with some more details would be the best way. Could you point them to https://www.mediawiki.org/wiki/How_to_report_a_bug ? [15:58:32] I will [16:04:47] marktraceur: if that was any of the uploadcampaigns that was definitely me [16:06:00] YuviPanda: it is the pikiwik one, iirc [16:06:09] marktraceur: yeah, but no idea what they were using. [16:06:17] marktraceur: do you know what pikiwiki does? [16:07:13] err [16:07:13] matanya: ^ [16:07:26] * matanya is looking [16:09:10] YuviPanda: can you look at the contrib of https://commons.wikimedia.org/wiki/Special:Contributions/Pikiwikisrael ? [16:15:54] matanya: looking [16:18:04] matanya: can't spot anything obvious from that [16:18:17] ok, i'll try to find their dev to file a bug [16:19:24] matanya: ty. If they were using UploadCampaigns, then it was my change that broke it, and I'll work with them to fix their tool [16:19:39] sure, thanks a lot [16:19:45] * Reedy subcontracts YuviPanda [16:19:46] I did go through possible affected parties and didn't find anyone who would have too much trouble, so I killed it [16:20:12] Reedy: it's an 'api breakage' in the sense two api methods were killed with that deployment :) So I guess I'm responsible [16:20:20] Reedy: I did check usage before that, one had 0 hits and one had ~60 [16:20:57] apergos, parent5446: hello [16:21:16] hey [16:21:26] Technical_13: Whatcha writing in JavaScript? :) [16:22:50] i wrote up the format specification: http://www.mediawiki.org/wiki/User:Svick/Incremental_dumps/File_format/Specification [16:23:12] sweet [16:24:09] Quick question: what is the subtype of lists? [16:24:26] parent5446: More context, please [16:24:43] since i'll have to have a version of the code that's almost useful for the mid-term, i think it makes sense to publicize the two at the same time: in about a week [16:24:45] Do you mean wikitext lists, HTML lists, MW API lists, Python lists, some other list [16:25:03] marktraceur: I mean incremental dump format lists. :/ [16:25:12] It says the type varies, but how is the type stored. [16:25:15] Ah, then I have no clue [16:25:30] marktraceur: that question was directed at me [16:25:30] marktraceur: we're doing our daily gsoc chat :-) [16:26:17] Overall though a good document to have. Makes it much easier to understand the file format. [16:26:32] * marktraceur suggests using tab-completion to highlight people [16:26:38] parent5446: that depends on the type in the list; if it's a list of 4-byte integers, each item is stored as 4 bytes; if it's a list of string, each item is stored as a normal string (length + string bytes) [16:27:10] Yes, but how do you determine the type? I'm guessing it just depends on the context? [16:27:37] I see lists used once in here only, list of 4 byte ids of revisions of this page [16:27:42] currently anyways [16:28:36] that's the same everywhere: how do you determine whether you should expect a list of integers; a single integer or a string? it always depends on the context [16:28:52] so list doesn't store its type [16:29:06] OK, that's what I figured. Just wanted to clarify. [16:29:18] ok [16:30:17] if the contributer name was deleted, what's done? same question for other fields I guess [16:30:57] where by deleted I mean the xlm tag contents would read "deleted" and only an admin or an oversighter or something would have access to it [16:31:30] yeah, i don't have any special handling for that yet [16:31:39] Are dumps supposed to have access controls? I mean I guess the final output would have it replaced, but if you're dumping as an oversight user... [16:31:48] dumps are public information [16:32:00] except the private tables which are dumped wholesale via myysqldump [16:32:02] In our case they are. [16:32:11] OK [16:32:48] The main reason I ask is that dumps can possibly be used for exporting and importing wikis, so if "deleted" is the user... [16:33:11] they are used for it and if you want things that are deleted (revisions and whatnot) you won't get them [16:33:20] OK [16:34:06] marktraceur: "hello world"... [16:34:06] I'm just wondering what the expected result would be. Let's say a revision's user only is deleted. When imported, the revision itself isn't deleted, so it's still imported, but the database needs to be in a valid state. [16:34:23] Taking an online class... [16:34:54] I don't know what import.php does with that [16:35:12] different scripts do different hacky things [16:35:24] people doing data analysis though want to know that it's actually a deleted user and not [16:35:37] oh this was a user with id 0 because of some bug [16:35:39] for example [16:35:52] OK, we'll leave that to the SpecialImport logic I guess. [16:36:17] as long as you have a way to mark it as 'deleted' in your format [16:36:31] and can produce the corresponding xml, then yep [16:36:35] yeah, i don't have that now; i will have to add that [16:36:48] Mhm [16:36:51] yep [16:37:38] so because there is an index and I guess the ids are unique in the index (for e.g. revisions) [16:37:52] ah rats, it's not dealign with the text id. right [16:38:52] if you had an index for the text id you could easily handle the cases where text id is duplicated, not storing a second copy of the text [16:39:02] I dunno if that would be a gain in the end or not [16:39:07] (in size) [16:40:44] mm there are missing fields it looks like too, rev_sha1 for example [16:41:00] how often does that happen? because it would require a new index for each text id, and that could be something like 2GB for enwiki 500M rev, most of them different text id, 4B per text id [16:41:11] I don't know how often [16:41:34] yeah, i know, there are bunch of fields missing [16:41:41] ok [16:41:45] as long as it's on your radar [16:42:28] fox example the content handler ones are missing too [16:44:06] oh, i think the XML schema has a bot flag for revisions, but AFAIK it's not actually used in the dump (probably because it's not actually saved), is that right? [16:44:22] I tried to do a check based on max of the revid and old_id on en wp but in fact there are a lot more text ids so that's a bust, we'd have to figure it out some other way [16:46:23] Yes, in the DB there's no info about it being a bot edit. [16:46:36] it's in rc [16:46:42] rc_bot I guess [16:46:54] Yes, but once it's gone from RC, then there's nothing. [16:46:56] :P [16:47:02] but not in any of the revision fields [16:47:57] it would have been nice for analysts but that's how it goes [16:48:00] hindsight etc [16:48:14] yeah, that's what i thought [16:48:59] If that plan the architecture team had to add rev_flags to the revision table ever pulls through, maybe we'll have bot flags... [16:49:28] but not for the first 500 million revs [16:49:43] I expect that's lower priority right about now [16:49:50] (compared to some other hot issues) [16:50:34] thanks for getting this up there on the wiki [16:51:00] you're welcome [16:51:27] so, what i'm working on now are indexes (making a tree out of them) [16:52:07] mm one other thing to think about, as we add fields, let's say that rev_flags really happens [16:52:14] or some other field gets added to all the pages, who knows [16:52:18] happens often enough [16:52:40] we would wind up rewriting the entire file in order to get those fields in I guess [16:52:44] rev_flags will probably happen once somebody does it. They were pretty adament about it b/c it solves a number of performance problems. [16:52:50] Krinkle: got some spare time atm? [16:53:16] yeah, as it is now, doing that would require rewriting the whole file [16:53:42] no [16:53:43] if we didn't want to do that, we would need to store some kind of version with each page/revision [16:53:44] if we are using these to regenerate fulls, that's a problem [16:54:04] Krinkle: k, would be nice if you could ping me once you're free for a bit [16:54:25] another thing to think about... [16:55:43] hoo: unlikely, contact me by means I can keep track of and respond when I can. [16:55:46] on the other hand, if each revision had a 1 byte revision number (not sure if smaller makes much sense), that feels like a waste of 500 MB to me [16:56:02] *version number [16:56:08] Krinkle: via mail? [16:56:21] or bugzilla, gerrit, depends. Yes. [16:56:39] it's going to be a size/speed tradeoff no matter what you do [16:56:39] k [16:56:43] but if every revision gets a new field added to it [16:56:59] eg the content model stuff, let's say that was new in the next schema [16:57:21] then it's a matter of where you put that stuff without rewriting each revision [16:57:26] yeah; also rewriting the whole dump shouldn't be that bad, since it wouldn't need to recompress revision texts [16:57:31] so some thinking about extensibility [16:57:57] that's true, no additional compression [16:59:33] in any case, each object already has an "object kind", which leaves some space to do this later on, even with the current format [17:00:25] for example, object kind 0x11 is page object v1, 0x21 could be page object v2+ [17:01:34] be good to get some discussion about it going on the list when you publicize it next week [17:02:12] ok, i can specifically ask about that [17:05:26] apergos, parent5446: anything else? i don't have anything else for today [17:06:15] Nope, but apergos, which one of us should do the midterm eval? [17:06:22] I'm idly thinking about Randall's last email (suggesting looking at things like sqlite) [17:06:31] wondering how horrid it woul dbe [17:06:41] parent5446: the primary mentor is supposed to do that [17:06:48] OK, I figured. Just checking. [17:06:52] I'm happy to consult with you, it's basically a questionnare [17:06:57] (did you see the email?) [17:07:04] Yeah I saw the questions. Pretty basic. [17:08:08] ok. so I'll be taking care of that next week [17:09:38] OK, well if that's all I'll see you both tomorrow. [17:09:43] I got nothing else [17:09:50] so yep, see everyone tomorrow [17:10:36] i think it makes sense to have a control over the file format, since we're dealing with so much data; for example i considered using something like delta compression of ids in indexes (since they often incremement just by 1); doing things like that would be pretty much impossible with something like SQLite [17:11:16] but i'll certainly look at the delta coding libraries that exist (when i get to that), that could make my work much easier [17:11:34] well one of the fun things about having a published spc is that folks can se what munging it a bit for some lightweight db would look like [17:11:40] folsk other than you, that is :-) [17:12:03] yeah, hopefully one of those is close enough to what you need [17:14:06] from reading the delta compression paper, it seemed to me those libraries should fit nicely with what i need [17:14:29] excellent [17:16:10] see you then [17:16:19] so I'm gonna get food, officially outa here [17:16:20] see ya [18:22:50] hi! what happened with ukrainian wikivoyage? All pages outputs: MediaWiki internal error. Exception caught inside exception handler. Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. [18:23:16] Wikidata deploy [18:23:18] rluts: currently being worked on in #wikimedia-wikidata [18:23:19] Just waiting for a script to run [18:23:20] rluts: Wikidata deployment related, working on [18:48:08] rluts: ukrainian wikivoyage should be fixed now [19:23:32] 'xcuse for interrupting the nice silence [19:23:34] If someone is interested in apparently full auto bot that writes ramblings with single spam link that is not in that SpamBlacklist feed from Wikimedia and bypasses our CAPTCHA.. If it bothered to it _could_ imo pass as a human if it wanted - http://develop.consumerium.org/wiki/Special:RecentChanges [19:23:58] I mean it seems to be an experiment in fully automatic text generation [19:24:26] I'm in the process of bannhammering it's nicks over past 2 weeks for a year [19:24:40] I'm fighting a losing battle [19:26:02] human registered accounts are maybe 100 accounts and according to http://wikiapiary.com/wiki/Consumerium there are 3.4k accounts and rising steadily so the bots are getting around the current CAPTCHA. Halp #wikimedia-tech ! [19:28:08] * jubo2 makes mental note to look into better CAPTCHA(s) .... does the one that Wikimedia uses for new account creations, does it hold the bots back or do they just laugh "haha" as they pass it in a few seconds ? [19:32:39] jubo2: Even Wikimedia's anti-spam measures do not stop many of them from registering. [19:33:23] This conversation should be taken to #mediawiki :) [19:33:46] Why register if you can edit anonomously? [19:41:36] it is not possible to translate in zh-hans in commons o-O [19:55:01] jubo2: well nowadays most spammers use things like decaptcha - http://decaptcha.biz/ and real people actually do the decaptchas [19:55:56] is it possible to enable zh-hant and zh-hans in the Commons Translate extension? [19:56:02] jubo2: so the idea of having a botproof captcha is false [20:02:17] RoanKattouw: around? [20:03:03] Yes [20:03:11] might this be a bug? https://nl.wikipedia.org/w/index.php?title=Help:Helpdesk&diff=prev&oldid=38546845 [20:05:33] akoopal: Waarschijnlijk wordt die "Zoek een artikel" link daar geplaatst door JavaScript (Gadget of MediaWiki:Common.js ofzo), en JS is niet toegestaan op de aanmeldpagina om veiligheidsredenen [20:06:12] Als ik JS uitschakel in m'n browser krijg ik hetzelfde gedrag op alle pagina's [20:06:44] tja, ik zit er totaal niet in, maar waarom komen deze berichten er dan onvertaald, en anderen niet? [20:06:51] Geen idee [20:06:56] Er lijkt JS te zijn die dit fixt [20:07:03] Maar waarom het niet gewoon werkt snap it ook niet [20:07:16] misschien een oude hack die is blijven steken ergens?\ [20:07:59] van voordat er goede internationalisatie was? [20:08:06] Misschien [20:08:11] Maar MediaWiki:Sidebar ziet er goed uit [20:08:58] hmm [20:11:00] BTW [20:11:18] Volgens MediaWiki:Sidebar is het "Vind een artikel" maar JS maakt er "Zoek een artikel" van [20:12:01] akoopal: OK, fixed [20:12:09] akoopal: Zie mijn recente edits in the MediaWiki-naamruimte [20:13:54] hmmm [20:14:07] Geen idee waarom het eerst niet werkte, maar het werkt nu [20:14:27] terugplaatsen van etalage snap ik niet helemaal, maar als het werkt ... [20:14:36] antwoord jij zelf op de helpdesk?\ [20:14:41] Sure [20:14:51] Zie ook https://nl.wikipedia.org/wiki/MediaWiki:Common.js , onderaan [20:15:01] * Tijdelijke oplossing voor het probleem dat een paar items in het linkermenu niet in het Nederlands worden weergegeven. http://nl.wikipedia.org/w/index.php?title=Wikipedia:De_kroeg&oldid=37918851#Heuh.3F.21_Engelstalige_linkwoorden [20:15:30] Hmm, snelcursus is nog stuk, fix ik die ook even [20:19:14] https://bugzilla.wikimedia.org/show_bug.cgi?id=46579 [20:19:22] die bug lijkt gerelateerd [21:08:20] hi. Who is good at mailing lists? [21:09:21] !ask [21:09:21] Hi, how can we help you? Just ask your question.