|
2018-11-14 10:53:18
|
<sqL_php_beginner>
|
something is going wrong in http://www.sub-bavaria.de/w/mw-config/index.php?page=Language ...... should rewrite localsettings manually?
|
|
2018-11-14 14:54:40
|
<alkamid>
|
Hi, I was told I can find people who know about database dumps here. I'm trying to read plwiktionary-...-pages-articles-multistream with Python (but I think it's a language-agnostic question). The first line of the index file is "670:1:czytać". But if I open the dump file like so: with open(bz2.BZ2File(path)) as f: and then f.seek(670), I'm not getting anywhere close to "czytać", which actually starts at 2730th byte. According to https://en
|
|
2018-11-14 14:54:40
|
<alkamid>
|
.wikipedia.org/wiki/Wikipedia:Database_download#Should_I_get_multistream?, " The first field of this index is # of bytes to seek into the archive"
|
|
2018-11-14 15:00:14
|
<wm-bot>
|
Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
|
|
2018-11-14 15:50:14
|
<wm-bot>
|
Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Thiemo_WMDE & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
|
|
2018-11-14 16:00:22
|
<CFisch_WMDE>
|
\o/
|
|
2018-11-14 16:00:40
|
<CFisch_WMDE>
|
Welcome to the Technical Advice IRC meeting!
|
|
2018-11-14 16:01:35
|
<CFisch_WMDE>
|
This time with /me and a still has to join Thiemo!
|
|
2018-11-14 16:02:29
|
<CFisch_WMDE>
|
There are no questions pre-posted on the wiki so just fire away!
|
|
2018-11-14 16:02:35
|
<CFisch_WMDE>
|
Hi Thiemo_WMDE :-)
|
|
2018-11-14 16:02:42
|
<Thiemo_WMDE>
|
Hi tech!
|
|
2018-11-14 16:04:03
|
<alkamid>
|
Hi CFisch_WMDE and Thiemo_WMDE! Can I start? It's a question about (poorly documented) multistream database dumps. I'm trying to read plwiktionary-...-pages-articles-multistream with Python (but I think it's a language-agnostic question). The first line of the index file is "670:1:czytać". But if I open the dump file like so: with open(bz2.BZ2File(path)) as f: and then f.seek(670), I'm not getting anywhere close to "czytać", which actually
|
|
2018-11-14 16:04:03
|
<alkamid>
|
starts at 2730th byte.
|
|
2018-11-14 16:04:21
|
<alkamid>
|
According to https://en.wikipedia.org/wiki/Wikipedia:Database_download#Should_I_get_multistream?, " The first field of this index is # of bytes to seek into the archive"
|
|
2018-11-14 16:05:26
|
<CFisch_WMDE>
|
hmm *looks*
|
|
2018-11-14 16:05:50
|
<d3r1ck>
|
o/
|
|
2018-11-14 16:06:32
|
<alkamid>
|
(see https://dumps.wikimedia.org/plwiktionary/20181101/ for the latest dump)
|
|
2018-11-14 16:07:32
|
<alkamid>
|
I have never done bz2 reading before so I might be completely wrong in my interpretation though!
|
|
2018-11-14 16:08:23
|
<CFisch_WMDE>
|
So you got the index and the multistream but when you try to access the position in the multistream where the index points you to you're not where you expect to be, right?
|
|
2018-11-14 16:09:00
|
<CFisch_WMDE>
|
I have never done bz2 reading before so I might be completely wrong in my interpretation though! <- Same here, but maybe we can solve this together ^^
|
|
2018-11-14 16:09:15
|
<CFisch_WMDE>
|
Or any volunteer that knows how to do that in the channel?
|
|
2018-11-14 16:09:16
|
<alkamid>
|
CFisch_WMDE, yes, thats correct
|
|
2018-11-14 16:09:23
|
<Lucas_WMDE>
|
does the offset refer to the compressed or decompressed stream?
|
|
2018-11-14 16:09:28
|
<Lucas_WMDE>
|
I suspect that might be the cause
|
|
2018-11-14 16:09:36
|
<CFisch_WMDE>
|
I just wanted to say the same
|
|
2018-11-14 16:09:38
|
<CFisch_WMDE>
|
:-)
|
|
2018-11-14 16:09:56
|
<CFisch_WMDE>
|
From what I read in on the wiki I would assume to the compressed stream
|
|
2018-11-14 16:10:26
|
<CFisch_WMDE>
|
but might also be the other way around
|
|
2018-11-14 16:11:00
|
<alkamid>
|
Oh, that would make sense
|
|
2018-11-14 16:11:16
|
<alkamid>
|
let me try
|
|
2018-11-14 16:18:43
|
<alkamid>
|
works like a charm, thanks a lot guys!
|
|
2018-11-14 16:19:11
|
<Lucas_WMDE>
|
I’m a bit surprised seeking into a compressed stream works, but okay :)
|
|
2018-11-14 16:19:49
|
<Lucas_WMDE>
|
alkamid: so how did you change your code? it would probably be nice to update the documentation to clarify this
|
|
2018-11-14 16:20:30
|
<CFisch_WMDE>
|
alkamid: you're welcome
|
|
2018-11-14 16:22:27
|
<CFisch_WMDE>
|
Anyone else? :-)
|
|
2018-11-14 16:26:37
|
<d3r1ck>
|
Thiemo_WMDE: Last meeting we had a discussion about the AutoloaderStructureTest
|
|
2018-11-14 16:27:36
|
<d3r1ck>
|
First approach was trying to fix the AutoloaderStructureTest itself but no luck yet
|
|
2018-11-14 16:28:18
|
<d3r1ck>
|
Looking for the pending patch right now....
|
|
2018-11-14 16:28:18
|
<Thiemo_WMDE>
|
I remember. You said you had 3 ideas.
|
|
2018-11-14 16:28:33
|
<d3r1ck>
|
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/472202
|
|
2018-11-14 16:28:54
|
<d3r1ck>
|
The 3 ideas came out as a result of the discussion of last meeting
|
|
2018-11-14 16:29:01
|
<alkamid>
|
Lucas_WMDE, I'm still figuring out what's the idiomatic way, but something along these lines: https://dpaste.de/LvUh
|
|
2018-11-14 16:29:02
|
<d3r1ck>
|
It is a bunch of suggestions, not from me :)
|
|
2018-11-14 16:29:39
|
<d3r1ck>
|
I was trying to understand how the class was written because any other way will be temporary
|
|
2018-11-14 16:30:02
|
<d3r1ck>
|
But solving the AST problem will be a permanent solution or say long term
|
|
2018-11-14 16:30:40
|
<alkamid>
|
I'm happy make a contribution to the documentation, but is that page (Wikipedia:Database_download) an appropriate place to put code samples? For me, those 4 lines in Python are more meaningful than the whole paragraph on that page...
|
|
2018-11-14 16:30:52
|
<d3r1ck>
|
Thiemo_WMDE: I'm wondering if you could just briefly, look at that class and tell me if you see something that can guide me fix it, I've tried
|
|
2018-11-14 16:32:54
|
<d3r1ck>
|
Reedy suggested that something is happening here: $expected = $wgAutoloadLocalClasses + $wgAutoloadClasses;
|
|
2018-11-14 16:33:07
|
<Lucas_WMDE>
|
alkamid: okay, so the offset refers to the compressed stream, not the decompressed contents?
|
|
2018-11-14 16:33:11
|
<d3r1ck>
|
But till now, I still can't see it :(
|
|
2018-11-14 16:33:18
|
<Lucas_WMDE>
|
(it wasn’t clear to me what your previous code did, so I couldn’t guess which way the fix went)
|
|
2018-11-14 16:33:24
|
<alkamid>
|
Lucas_WMDE, correct
|
|
2018-11-14 16:33:28
|
<Lucas_WMDE>
|
ok thanks
|
|
2018-11-14 16:33:55
|
<Lucas_WMDE>
|
ah, but I don’t have permission to edit the Wikipedia documentation page to add that clarification :D
|
|
2018-11-14 16:34:04
|
<Lucas_WMDE>
|
I’ll leave a message on the talk page
|
|
2018-11-14 16:35:06
|
<Thiemo_WMDE>
|
d3r1ck: Where is the patch with the failing test?
|
|
2018-11-14 16:35:11
|
<d3r1ck>
|
So the currently patch doesn't solve the problem though it makes sense, still digging to track down what is the issue honestly
|
|
2018-11-14 16:35:19
|
<d3r1ck>
|
looks...
|
|
2018-11-14 16:35:33
|
<d3r1ck>
|
Thiemo_WMDE: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GuidedTour/+/466675
|
|
2018-11-14 16:35:49
|
<apergos>
|
woops
|
|
2018-11-14 16:35:56
|
<apergos>
|
I was here and not paying attention because work
|
|
2018-11-14 16:36:44
|
<d3r1ck>
|
apergos: o/
|
|
2018-11-14 16:37:06
|
<Thiemo_WMDE>
|
Did you tried splitting the alias to a separate file named "GuidedTourLauncher.php"?
|
|
2018-11-14 16:37:15
|
<apergos>
|
the whole poin of hving the multistream index is to be able to seek into the compressed file
|
|
2018-11-14 16:37:21
|
<apergos>
|
which is fast
|
|
2018-11-14 16:37:35
|
<apergos>
|
seekinginto the uncompressed stream means you hve to decompress everything before it (slow)
|
|
2018-11-14 16:37:36
|
<d3r1ck>
|
Thiemo_WMDE: No, I've not tried that yet, so far, I've been trying option 1
|
|
2018-11-14 16:37:49
|
<d3r1ck>
|
Which is to fix the AST
|
|
2018-11-14 16:38:10
|
<d3r1ck>
|
I should not use AST again :(, it misleading
|
|
2018-11-14 16:38:11
|
<apergos>
|
once you're there you uncompress the block and you may have multiple pages in the block, but this is ok, dealing with one block is pretty quick
|
|
2018-11-14 16:38:12
|
<Thiemo_WMDE>
|
d3r1ck: Do you mind if I upload a PS9 to that patch?
|
|
2018-11-14 16:38:27
|
<d3r1ck>
|
Thiemo_WMDE: Please feel free, I don't mind :)
|
|
2018-11-14 16:38:31
|
<apergos>
|
checks out again for a bit, sorry but I am in the middle of sorting out the last part of a script
|
|
2018-11-14 16:42:29
|
<apergos>
|
fixed, back
|
|
2018-11-14 16:44:55
|
<Thiemo_WMDE>
|
d3r1ck: Done, lets see if this works better https://gerrit.wikimedia.org/r/466675
|
|
2018-11-14 16:45:22
|
<d3r1ck>
|
Okay!
|
|
2018-11-14 16:46:25
|
<apergos>
|
so all of that was to alkamid but it seems you got it all worked out anyways
|
|
2018-11-14 16:48:36
|
<d3r1ck>
|
Thiemo_WMDE: Is this normal? https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/7925/console
|
|
2018-11-14 16:48:50
|
<d3r1ck>
|
That report was available but on checking again later (today) it's not there :(
|
|
2018-11-14 16:49:01
|
<CFisch_WMDE>
|
apergos: still: thanks for your input in that ;-)
|
|
2018-11-14 16:49:34
|
<apergos>
|
sure, as the writer of that poorly maintained doc (and code, heh), figured it ws ok to chime in
|
|
2018-11-14 16:49:51
|
<CFisch_WMDE>
|
hehe
|
|
2018-11-14 16:51:47
|
<Thiemo_WMDE>
|
d3r1ck: Yea, these CI reports get deleted after a few weeks.
|
|
2018-11-14 16:52:20
|
<d3r1ck>
|
Okay! Now I know :)
|
|
2018-11-14 16:52:25
|
<d3r1ck>
|
Thanks!
|
|
2018-11-14 16:52:26
|
<Thiemo_WMDE>
|
You can retrigger the tests by writing a comment with nothing but "recheck" on the patch on Gerrit.
|
|
2018-11-14 16:52:40
|
<CFisch_WMDE>
|
apergos: But then you might be able to improve that documentation ... see request https://en.wikipedia.org/wiki/Wikipedia_talk:Database_download#Clarify_multistream_description ;-D
|
|
2018-11-14 16:53:11
|
<d3r1ck>
|
Thiemo_WMDE: Yeah, was trying to access that particular report, wanted to see something, it seems it was different since it was a different PS
|
|
2018-11-14 16:53:18
|
<apergos>
|
oh heh I maintain the 'official' (:-P) docs
|
|
2018-11-14 16:53:24
|
<apergos>
|
not the stuff on any given project about it
|
|
2018-11-14 16:53:26
|
<apergos>
|
um
|
|
2018-11-14 16:53:54
|
<d3r1ck>
|
Thiemo_WMDE: Things are a little different now: https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/10150/console
|
|
2018-11-14 16:54:58
|
<CFisch_WMDE>
|
ahhh :-) k
|
|
2018-11-14 16:56:05
|
<apergos>
|
well I added one word to the page, they probably won't mind :-D
|
|
2018-11-14 16:59:28
|
<apergos>
|
do I really not have that format documented anywhere... hmm maybe i's in a readme only, that would be unfortunate
|
|
2018-11-14 17:03:25
|
<apergos>
|
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dumps/mwbzutils/+/master/xmldumps-backup/mwbzutils/README meh
|
|
2018-11-14 17:03:36
|
<apergos>
|
documented in the util used to dump an arbitrary block
|
|
2018-11-14 17:03:43
|
<apergos>
|
remind me to fix that at some pint
|
|
2018-11-14 17:03:45
|
<apergos>
|
*point
|
|
2018-11-14 17:04:14
|
<CFisch_WMDE>
|
Ok the official part of the Technical Advice IRC meeting is done for the week - see you in one of the next sessions or online :-)!
|
|
2018-11-14 17:04:48
|
<CFisch_WMDE>
|
And don't forget: https://discourse-mediawiki.wmflabs.org/ is open 24/7 for your questions ;-)
|
|
2018-11-14 17:05:06
|
<bawolff>
|
As always, thank you for doing these :)
|
|
2018-11-14 17:05:38
|
<CFisch_WMDE>
|
:-)
|
|
2018-11-14 17:08:53
|
<revi>
|
meanwhile... anyone familiar with MW-Vagrant?
|
|
2018-11-14 17:09:21
|
<revi>
|
I know advice stuff ended just around 10 minutes ago but 10 minutes ago I was not getting problem >_>
|
|
2018-11-14 17:11:26
|
<revi>
|
I did `vagrant up`, and then I did `vagrant roles enable checkuser` and what I receive is 'WARNING: This has been deprecated in favor of `vagrant cloud auth login` which looks unrelated for MW-Vagrant
|
|
2018-11-14 17:12:34
|
<revi>
|
vagrant version 2.2.0
|
|
2018-11-14 17:16:30
|
<revi>
|
SleepyOne: ^ :-p
|
|
2018-11-14 17:19:33
|
<revi>
|
did `vagrant provision`, failed, did roles enable, 'Ok' this time
|
|
2018-11-14 17:19:36
|
<revi>
|
lol huh
|
|
2018-11-14 17:44:16
|
<alkamid>
|
apergos, thanks for multistream clarifications!
|
|
2018-11-14 17:44:28
|
<apergos>
|
sure!
|
|
2018-11-14 17:45:14
|
<apergos>
|
it seems I should write an 'output file formats' page someplace and link to it prominently from either meta or wikitech
|
|
2018-11-14 22:45:03
|
<ragesoss>
|
@tgr: could you approve this updated dashboard.wikiedu.org consumer? It's the same as the previous consumer except that it adds the high-volume editing grant. https://meta.wikimedia.org/wiki/Special:OAuthListConsumers/view/b20ec7f1dae4fab37c83fc30aea07bad
|
|
2018-11-14 22:45:24
|
<ragesoss>
|
(This matches what outreachdashboard.wmflabs.org already has: https://meta.wikimedia.org/w/index.php?title=Special:OAuthListConsumers/view/5709c54e5e241577730e27c13e1a56cf&name=&publisher=Ragesoss&stage=1 )
|
|
2018-11-14 22:46:39
|
<ragesoss>
|
We're enabling the same account creation feature on Wiki Education Dashboard that we added to Programs & Events Dashboard earlier this year.
|
|
2018-11-14 22:55:41
|
<andre__>
|
ragesoss: Hej! On an unrelated note, are the two unpublished GCI tasks (Capybara feature tests; no-undef eslint rule) ready to be published, or do they need more tuning? (I'm unable to tell, unfortunately)
|
|
2018-11-14 23:06:17
|
<ragesoss>
|
andre__ they're ready. Should I have done something to signal that?
|
|
2018-11-14 23:09:50
|
<ragesoss>
|
Bryan approved the consumer already, tgr.
|
|
2018-11-14 23:18:42
|
<andre__>
|
ragesoss: Ah, thanks, will publish. For future reference, feel very free to add a "[READY TO PUBLISH] " prefix or such :)
|
|
2018-11-14 23:18:56
|
<andre__>
|
(For all those years I've never come up with a better solution, meh.)
|