[08:28:12] aude: you there? [08:28:19] Reedy: you there? [08:48:16] Denny_WMDE: ? [08:48:26] Surprisingly I am.. [08:48:29] the dispatcher seems to have stopped working yesterday [08:48:51] i am just looking at the stats, and it is not moving, and seems to has not since yesterday evening [08:49:03] https://www.wikidata.org/wiki/Special:DispatchStats [08:49:31] i don't know what got changed, or what happened, but this means that all updates to the wikipedias are currently stuck [08:49:40] i saw reports on that from the community [08:49:51] i am not sure how to proceed [08:50:11] I'll look at the logs for it [08:50:15] thank you [08:50:28] Why is there wikibase-dispatch-changes and wikibase-dispatch-changes2? [08:50:30] :/ [08:50:34] They look identicle [08:50:40] identical [08:50:49] i don't know, sorry. first time i hear of that. [08:51:01] aude and danielk are both not here right now. [08:52:14] Denny_WMDE: It keeps fatal-ing [08:52:15] Fatal error: Cannot use object of type Diff\DiffOpChange as array in /usr/local/apache/common-local/php-1.22wmf9/extensions/Wikibase/lib/maintenance/dispatchChanges.php on line 858 [08:52:40] what? [08:52:49] has there been a deployment yesterday? [08:53:46] Not obviously [08:53:47] I did a wikidata folder sync on the 11th [08:54:03] yes, but the errors started on the 12th? [08:54:18] i am checking the code at that place [08:54:49] Need to uncompress some log files to check [08:57:39] Denny_WMDE: Yup. Was fine at the start of the 12th, not by the end [08:57:47] thank you [08:58:03] so it keeps throwing these fatals right now? [08:58:34] Yup [08:58:52] hmm, i have an idea about the culprit [08:59:55] could someone else have made a deploy yesterday? that looks like code that was deployed to stop the aliases of being deleted [09:00:18] https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:00:34] I can't see a scap/full code tree sync [09:01:15] Started at 171252 [09:01:24] Uhh, that's a line number [09:01:45] ?? [09:02:39] Hmm [09:02:40] 15:09:18 Posted 200 changes to snwiki, up to ID 55754436, timestamp 20130712150916. Lag is 2 seconds. Next ID is 55754436. [09:02:40] Fatal error: Cannot use object of type Diff\DiffOpChange as array in /usr/local/apache/common-local/php-1.22wmf9/extensions/Wikibase/lib/maintenance/dispatchChanges.php on line 858 [09:02:40] 15:09:19 Posted 200 changes to diqwiki, up to ID 55754366, timestamp 20130712150907. Lag is 12 seconds. Next ID is 55754366. [09:03:14] Then there was a few more working [09:03:17] then it all died [09:03:19] 15:10 or so [09:03:40] 15:20 Krinkle: Resetting commons:User:CommonsDelinker account e-mailaddress (bug 51016) [09:03:40] 10:43 mark: Sending European wikidata and wikivoyage traffic to esams [09:03:42] Neither are relevant [09:03:56] no [09:04:17] i am trying to figure out a few things [09:09:58] Denny_WMDE: Ooh. I see a profiling bug :p [09:10:01] weird, that diff looks pretty normal [09:10:09] a profiling bug? [09:10:23] Missing wfProfileOut [09:10:25] I'll fix and commit [09:10:26] ah it does not profileout [09:10:30] ok thanks [09:10:42] but that's not the culprit, right? [09:12:03] I hope not! [09:12:54] it is this change it is choking on right now http://www.wikidata.org/w/index.php?title=Q11076586&diff=prev&oldid=55754436 [09:13:00] which does not make any sense [09:13:16] but we were working yesterday on some changes to the aliases [09:14:10] that might have changed some diffops, but there was no plan to deploy that afaik [09:14:29] New patchset: Reedy; "Add missing wfProfileOut() call" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73562 [09:14:52] Change merged: Denny Vrandecic; [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73562 [09:21:45] the involved code has been changed in march resp. in may the last time [09:21:50] :-/ [09:23:57] The comment is also seemingly wrong.. * @return \Diff\MapDiff|bool [09:24:06] Bug getting Diff\DiffOpChang [09:24:09] *But [09:26:02] yes, i am looking into this [09:32:36] meh, i have a stupid idea which doesn't fix anything probably, so be free to say no — is it possible to simply bump by this change and go to the next? just see if it is choking on this diff for some reason, or if something else is going on [09:33:10] Reedy: ^ [09:33:20] JeroenDeDauw: ping [09:33:29] Presuambly.. [09:33:34] Where does it need removing from or whatever? [09:33:58] the queue for each wiki needs to be bumped by 1 [09:35:25] UPDATE wb_changes_dispatch SET chd_seen = chd_seen +1; [09:35:26] ? [09:36:09] uhmmm… (blush) [09:36:13] i do not know... [09:36:26] chd_seen INT NOT NULL DEFAULT 0, -- last change ID examined (dispatch state) [09:36:33] sounds good [09:36:46] i would try... [09:36:55] Query OK, 287 rows affected (0.00 sec) [09:36:55] Rows matched: 287 Changed: 287 Warnings: 0 [09:36:57] and see if the fatals change accordingly, or if they even dsisappear [09:36:59] sounds good [09:37:34] Guess we might have to wait upto 5 minutes or so [09:37:43] ok [09:37:55] well, just over 2 minutes based on the current time [09:39:14] they are bumped up [09:40:09] 09:40:01 Starting loop for unlimited passes or 900 seconds [09:40:09] Fatal error: Cannot use object of type Diff\DiffOpChange as array in /usr/local/apache/common-local/php-1.22wmf9/extensions/Wikibase/lib/maintenance/dispatchChanges.php on line 858 [09:40:35] the last idea is to bump it up by 5000 [09:40:48] I can try that [09:42:26] Ahh [09:42:33] Done that, now running it manually.. [09:42:48] Over 66000 seconds of lag [09:42:55] so it runs again? [09:42:58] yup [09:43:37] ok i am noting down the number range [09:43:41] of diffs [09:43:55] we have nailed it down to 5000 diffs, can't be that hard to find the relevant one… :) [09:44:13] heh [09:44:24] thank you very much for your help, let's see if it chokes again or if it catches up now [09:45:59] We could pipe tail -f /var/log/wikidata/dispatcher2.log into here ;) [09:47:22] I'll come back and check on it in a while [09:49:41] :) [09:49:44] thanks [10:13:07] Change on 12mediawiki a page Extension:Wikibase Client was modified, changed by 90.206.178.170 link https://www.mediawiki.org/w/index.php?diff=732312 edit summary: [+126] /* Configuration */ [10:54:39] New patchset: Denny Vrandecic; "Check if Diff is a value before accessing value field" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73566 [11:00:58] New patchset: Denny Vrandecic; "Check if Diff is a value before accessing value field" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73566 [11:01:24] New patchset: Denny Vrandecic; "Check if Diff is an array before accessing field" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73566 [11:41:39] aude: can you help me? [11:47:24] Hey, ich suche einen Link auf den Monatsplan den Lydia vor kurzem herumgeschickt hat. [11:51:26] BenutzerConny: dieser? http://www.mediawiki.org/wiki/Roadmap#Wikidata_deployment [11:51:59] Gefunden :) http://www.mediawiki.org/wiki/Roadmap#Wikidata_deployment [12:15:12] Heh, Denny, it turns out I'm not very useful on the weekends when teathering xD [12:20:57] hai addweekend [12:21:03] Change on 12mediawiki a page Extension:Wikibase Client was modified, changed by 90.217.114.74 link https://www.mediawiki.org/w/index.php?diff=732332 edit summary: [+165] [12:21:36] Conny was here? :O [12:25:46] Hiya vogone [12:26:39] Vogone: Was :/ [12:27:17] addweekend: are you currently in Berlin? [12:28:05] Yus [12:29:27] superb [12:29:33] k [12:29:38] wait a sec [12:29:44] I'll come over [12:30:26] addweekend: Run, I'll book you tickets for the first flight back to the UK if want to escape Vogone :P [12:30:49] XD [12:31:17] :( [12:32:15] are you always in this tiny village? [12:32:33] or do you travel sometimes in Germany? [13:19:55] anyone know if anyone is working on image links in wikidata, and global usage? [13:20:37] it is going to be problematic with file replacement at Commons as the wikidata link doesn't show up, yet the wikis call WD component [13:38:18] sDrewth: good point, thanks, noting it down [13:40:22] Denny_WMDE: added it to the project chat [13:40:49] thanks [13:42:43] Denny_WMDE: Shall we backport the is array patch? [13:43:39] i am vary of backporting anything right now due to the dispatch error [13:43:44] Change merged: jenkins-bot; [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73566 [13:43:48] so it was not backported yet, good to know [13:43:56] hoo: how does the bug show? [13:45:12] What do you mean? [13:45:30] But this will solve the issue in case the var really is an object [13:47:42] Looks like we're almost caught back up [13:48:04] well, bar bpywiki ;) [13:49:46] Reedy: So you've just restarted the thing and it's fine ever since? [13:50:01] I'm confused on how it works without the patch :P [13:51:15] No [13:51:37] chd_seen was increased by 5001 on all wikis [13:52:09] and then it just started working again? :P Weird, but ok :) [13:52:56] Denny_WMDE: I guess this wouldn't have happened with job only dispatching... :P [13:53:52] sorry, misunderstood [13:54:02] i thought you were talking about yesterdays array / object bug [13:54:08] yes, we could backport this [13:54:13] it should take care of the problem [13:54:24] and it should backport without any interference [13:54:58] Yeah... but it's a workaround only... how on earth did that even happen [13:55:08] hoo: no idea actually [13:55:26] there is probably one edit there that was weird [13:55:41] Denny_WMDE: Maybe it was the alias changes stuff [13:55:42] yes, the dispatcher is doing a little bit too much… i was a bit surprised to see the size of the file alone [13:55:48] that i what i thought [13:55:54] that has recently been changed a few times (with backports AFAIK) [13:55:59] but there have been more alias changes since then i am pretty sure [13:56:18] Maybe it's only causing trouble in few cases [13:56:22] it seems so [13:56:33] and I think they had one fix and reverted it soon after... [13:56:36] aude: ^ [13:56:37] i hope in very very few cases [13:56:50] yeah, addweekend or aude could know [13:56:55] but nothing was deployed yesterday [13:57:19] * Reedy wonders if the community will find the broken entry before we do [13:57:38] they have more eyeballs :) [13:57:52] It wouldn't be the first time either! [13:58:13] aye! [13:58:21] ah, the first wikis are catching up [13:59:06] https://github.com/wikimedia/mediawiki-extensions-Wikibase/commit/92baf158a3f0a02c4bcc2ccb4e0ecc515d1bfa96 [13:59:19] is introducing a new op change [13:59:28] AND it has been backported, so... [14:00:09] it could be [14:00:25] i am surprised it doesnt hit more often [14:00:37] i have no idea how to deploy the array-fix [14:00:44] but we could do it [14:01:01] Denny_WMDE: Maybe we should do wfLogWarning in case it'S non array [14:01:10] to not silently suppress errors? [14:01:13] sounds good [14:01:19] Will you do that? [14:01:45] i was just trying to fix it quickly in case we need it :) I am not a real developer, obviously [14:01:54] heck i studied philosophy, no php [14:01:59] Sure :) [14:02:03] :D [14:02:11] would you mind to do it? [14:02:31] That's a minor thing, sure ;) [14:02:38] thx [14:05:31] * hoo thinks of a good error message [14:05:47] wfLogWarning( 'ffs this should be an array' ); :P [14:06:37] Y U NO ARRAY [14:06:54] "this should never happen" [14:09:12] New patchset: Hoo man; "Log a warning if ItemChange::getSiteLinkDiff returns a non-array" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73573 [14:10:23] what the hell [14:10:59] Change merged: Denny Vrandecic; [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73573 [14:11:16] hoo: ? [14:11:32] @return \Diff\MapDiff|bool [14:11:38] how could that ever return an array [14:12:00] yes, Reedy and I were wondering the same... [14:12:23] I think, I see [14:12:28] there's something to say about static type checking [14:12:29] Well, "array" [14:12:53] class Diff extends \ArrayObject implements IDiff { [14:13:20] "This class allows objects to work as arrays." [14:13:25] class ArrayObject implements IteratorAggregate, Traversable, ArrayAccess, Serializable, Countable { [14:13:27] gj PHP [14:14:25] ouch [14:14:25] is_array isn't returning true for those [14:14:40] so our fixes will skip ALL entries [14:15:10] Denny_WMDE: ^ [14:15:25] Why would one want an object that can act as array [14:15:34] aaah [14:15:38] If it's like a wrapper [14:16:19] good thing we didnt deploy [14:16:25] and we are not deploying anything here without tests [14:16:42] New patchset: Denny Vrandecic; "Revert "Log a warning if ItemChange::getSiteLinkDiff returns a non-array"" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73574 [14:17:42] how comes i am the author and owner of that patch set? [14:17:50] Denny_WMDE: It's a revert [14:17:57] ahthanks [14:18:16] Change merged: Denny Vrandecic; [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73574 [14:18:38] New patchset: Denny Vrandecic; "Revert "Check if Diff is an array before accessing field"" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73575 [14:19:16] i am killing both patchsets now [14:19:35] all wikis have caught up [14:20:11] Ok :) [14:20:13] Why did one stay really out of sync till nearly the end? [14:20:31] Shouldn't it be weighted to maybe do the more lagged ones first [14:20:37] We do something similar on the job queue [14:21:09] It should, but I wouldn't recommend spending any more time on this [14:21:10] Reedy: need to ask DanielK [14:21:21] i agree with hoo [14:21:38] yeah, just seemed weird 1 was essentially still mostly lagged, then apparently caught up at hte end [14:21:51] Change merged: jenkins-bot; [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/73575 [14:22:09] if ( !$siteLinkDiff instanceof ArrayObject ) {...} [14:22:16] ^ we should probably do that [14:22:29] stupid is_array() -.- [14:22:33] i think it happens when a wiki is very "sparse" compared to wikidata [14:22:42] so that a lot of changes actually have no effect at all [14:23:00] so we should always see something there that has less than a 1000 articles or so [14:23:01] Guess it's not a big deal [14:23:08] never italian or english [14:23:28] hoo you have the propagation set up yourself? [14:23:39] for testing? [14:23:50] I nearly never run the script [14:23:59] For testing I usually just purge the client pages :P [14:24:16] but yes, it's working [14:28:28] the script is working? [14:28:31] or the purging? [14:28:47] both, hopefully [14:28:50] :D [14:29:06] could you do me a favour, and check whether the is_array indeed would not have worked? [14:29:23] already did: [14:29:24] php > $f = new ArrayObject(); [14:29:24] php > echo is_array( $f) ; [14:29:31] false [14:29:54] but you can happily use it as array and the isset() doesn't fatal out either [14:30:09] I suppose it isn't actually an array [14:30:22] Reedy: Sure, it's not... but it can be used as array [14:30:26] phpyeah [14:30:39] we should have is_array_compatible() [14:30:50] Hey look, there's a bug for it [14:30:54] hoo: https://bugs.php.net/bug.php?id=36930 [14:31:15] Ah [14:31:16] if(is_array((array) $arrayObj)) { [14:31:21] Try casting it to an array :) [14:32:22] Well, you can cast almost everything to an array [14:32:22] var_dump(is_array((array) 'meh...php')); [14:32:39] Sure [14:32:42] var_dump(is_array((array) new stdClass())); [14:32:45] even that is true [14:32:53] But are we going to be returning everything else? [14:32:58] $testResult = (is_array($testObject) || $testObject instanceof ArrayAccess); [14:33:05] it's a wonfix [14:33:28] We could introduce a wfCanBeUsedAsArray :P *hides* [14:33:44] wfIsThisPossiblyLikelyMaybeAnArraySometimes() [14:34:05] rather wtfIsThisPossiblyLikelyMaybeAnArraySometimes() [14:34:48] hoo: We could write some unit tests [14:34:57] testDoesPhpActuallyKnowWhatThisIs [14:35:21] SIGH [14:35:54] this wouldn't have happend with C or Cpp :P [14:36:10] Yeaaah [14:36:22] Having a compiler just say lolno is easier [14:36:40] Yes... and segfaults are nicer than fatals either [14:37:09] Well... PHP can segfault as well... so we got the best of both worlds :D [14:38:45] PHP just does whatever it wants [14:40:58] Well, in fact you can even make PHP use segaults only: Just register an own error handler, let if find the current PID and then send signal 11 to it... profit? [14:42:13] * hoo sends himself into programmers' madhous [15:05:53] *scrolls up* [15:14:09] I highly doubt the patch me and aude had for aliases has caused this, after all it was nowhere near anything to do with dispatch, [15:30:00] New patchset: Jeroen De Dauw; "Get rid of some global scope assumptions in repo config" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/72938 [16:56:39] who may I disturb for the Wikivoyage interwikis? [17:12:38] New patchset: Jeroen De Dauw; "Get rid of some global scope assumptions in repo config" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/72938 [17:18:54] New review: Hoo man; "Looks good now :)" [mediawiki/extensions/Wikibase] (master) C: 2; - https://gerrit.wikimedia.org/r/72938 [17:19:37] what about them, Sannita? [17:19:45] here for another 5 minutes or so, though [17:20:12] Change merged: jenkins-bot; [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/72938 [17:38:49] JeroenDeDauw: got a while? [17:39:14] (a little while) [17:39:30] lazowik: sure [17:39:45] lazowik: but only if you install my chrome extension :D [17:39:52] :D [17:39:55] ok, why not [17:40:55] installed, works as advertised ^^ [17:41:46] so [17:42:01] there's a $data array in Item data model [17:42:30] and now for sitelinks it looks like $data['links'][siteId] = pageName [17:42:44] and it's directly passed to serialization function [17:43:04] so I think I've got only two options: [17:44:52] replace pageName with array (name => pageName, badges => array of badges) and modify cleanStructure to modify data pulled from database [17:45:07] or add $data['badges'][siteId] = array of badges [17:45:30] what should I do? [17:45:30] or is there another way? [17:45:30] JeroenDeDauw: ^ [17:46:17] I think that the first option uses cleanStructure not in the way it should be [17:46:22] used [17:46:41] . [17:46:52] and breaking the serlialization seams to be messy [17:47:18] so is there a better way than 2? [17:47:44] lazowik: I was actually thinking about this yesterday - and those are the two most straight forward approaches [17:48:04] The thing with the serialization is that all the formats that we used at any point will need to be supported forever [17:48:13] yup [17:48:16] So we should be very careful when changing this or adding new stuff [17:48:55] also at one point I thought about something like $data['links'][siteIdBadges] [17:49:08] e.g. $data['links']['enwikiBadges'] [17:49:50] it would keep the first dimension clean [17:50:47] I suggest doing what you wrote about first. Changing the string element in the array to be an array with (name => pageName, badges => array of badges) [17:50:55] otherwise it would be label, description, aliases, claims, links and WHOOOPS badges [17:51:36] then where should change deserialization data? [17:51:47] *deserialized old data [17:52:37] and that code would stay forever there [17:52:56] getSimpleSiteLink(s) and addSimpleSiteLink probably [17:53:56] will try [17:53:59] thanks [17:54:09] off to eat something