[00:32:47] web upgrade has always been a bit of a crapshoot. [00:36:16] I uploaded a fix [00:37:00] on to the next issue, which is that Cindy pointed out that sqlite and postgres are showing an anomalous spike in pingback data https://pingback.wmflabs.org/#database-type [00:39:59] and indeed I am seeing database query errors from upsert(), not the actual bug yet but it is probably related [00:41:58] Either that, or someone has managed to get huge penetration of MediaWiki-on-MSSQL [00:41:59] :) [00:44:31] our number of installs per month jumped by a factor of 30 or so, we should crack open the champagne [00:47:20] UPDATE /* Pingback::markSent 127.0.0.1 */ updatelog SET ul_value = '1533083460' WHERE ((Pingback-1.32.0-alpha IS NULL)) [00:49:36] seriously [00:52:51] the code is from https://gerrit.wikimedia.org/r/c/mediawiki/core/+/419506/10/includes/Pingback.php CindyCicaleseWMF legoktm [00:53:33] the second parameter to upsert() is a row array, not a single row [00:57:23] the third parameter is an index list, it's also supposed to have numeric keys [00:58:19] ah, except for in the documentation in which the second parameter can in fact be a single row [00:58:36] #comment-dont-run [00:59:03] One day, we'll have AIs that run comments as code. [01:00:12] it is the third parameter that's the issue, that's where the WHERE clause comes from in the query I quoted [01:36:45] With our IRC ad service you can reach a global audience of entrepreneurs and fentanyl addicts with extraordinary engagement rates! https://williampitcock.com/ [01:36:45] I thought you guys might be interested in this blog by freenode staff member Bryan 'kloeri' Ostergaard https://bryanostergaard.com/ [01:36:45] Read what IRC investigative journalists have uncovered on the freenode pedophilia scandal https://encyclopediadramatica.rs/Freenodegate [01:36:49] A fascinating blog by freenode staff member Matthew 'mst' Trout https://MattSTrout.com/ [01:40:46] T200861 and T200864 for folks playing at home, both have patches uploaded now [01:40:47] T200864: Pingback on non-MySQL databases fails to save to updatelog, generates a high rate of unique pings - https://phabricator.wikimedia.org/T200864 [01:40:47] T200861: Web upgrade of SQLite does not work, just skips to install - https://phabricator.wikimedia.org/T200861 [01:51:06] TimStarling: Thank you for finding the pingback issue! You're awesome! [02:11:39] TimStarling: do we need to backport the sqlite change? [02:40:24] yes [04:16:41] TimStarling: uhoh :( we should also fix the abstraction layer so the same parameters work on all DBMSs? [04:17:54] MySQL wasn't using the index list because it doesn't need it [04:18:17] but maybe we could have a validation mode, for unit tests or something, in which mysql checks the index list? [04:19:09] hm yeah [04:19:13] and we should have tests for Pingback [04:44:25] TimStarling: also, a question for you on https://phabricator.wikimedia.org/T199511#4424435 [05:09:48] thank you :) [09:39:20] DanielK_WMDE: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/445200 is ready for review [10:07:49] tgr: thanks! I'll have a look later today, I have a buch of meetings now. [10:08:12] i wrote two minor comments when looking at the code yesterday, but forgot to send them. did that now [10:22:58] tgr: could you review this? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/443985 [11:04:52] DanielK_WMDE: yeah, that and the service manipulator patch are the next on my list [16:19:20] tgr: is this complete? https://phabricator.wikimedia.org/T200568 [16:20:37] DanielK_WMDE: AFAIK yes [16:23:01] tgr: ok, i'll close it, then [16:38:04] DanielK_WMDE: anything for SoS? write-both goes to production this week, right? [16:47:06] tgr: yes, and we'll start running the migration script [16:47:13] "we" is Tim, in this case [16:50:57] CindyCicaleseWMF: ^ edited the SoS text a bit, not sure if it is worth a callout [16:51:56] tgr: thanks! [18:44:27] Hm. curious why phpunit is not considering some of the Language classes as code, but does others [18:44:28] https://doc.wikimedia.org/cover/mediawiki-core/languages/classes/LanguageAr.php.html [18:44:32] Is it because of @todo? [21:19:19] Krinkle: hmm, @todo really shouldn't do that :/ [21:20:49] legoktm: Yeah, I can't find why it happens. [21:20:59] Might not be caused by @todo, but if not that, then what? [22:18:29] Krinkle: want to review+merge https://gerrit.wikimedia.org/r/q/topic:sqlite-postgress-pass ? I'd like to enable sqlite/postgres tests as voting to prevent further regressions [22:23:00] ty :) [22:23:05] * legoktm updates zuul config [22:23:33] * Krinkle checks for travis build to become green and make that voting, too [22:40:17] TimStarling: Amended per comments - https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/447016/ and stack [23:03:19] legoktm: When an exception happens on-wiki printed by OutputPage nicely (red box, exception id), you'd expect to find that in Logstash, right? [23:03:43] Krinkle: yes... [23:03:52] I find as of late, while there are errors in that channel, anything I get myself or from phabricator reports, is never there. I do find it on mwlog1001 /exception.log but never in Logstash [23:03:55] Not sure what's going on.. [23:04:17] tbh I rarely use logstash, always using mwlog1001 [23:04:52] I mean, like https://en.wikipedia.org/w/index.php?title=Talk:C_Real&action=edit&redlink=1 for example shows a fatal for me. [23:05:06] exception id 'W2I8mApAICMAAILYTWkAAABW' [23:05:49] But doesn't show on Logstash. Does show on mwlog. [23:06:02] Looking in logstash for `type:mediawiki AND channel:exception AND wiki:enwiki` I don't see other ones either (last 30min) [23:06:20] except for an unrelated DBQueryTimeoutError entry [23:40:48] I've never really dug into logstash but it seems to be well documented on wikitech [23:48:58] hmm, it could do a bit better explaining how messages get from MW to logstash [23:49:12] this diagram is incorrect: https://wikitech.wikimedia.org/wiki/Logstash#/media/File:Wmf-elk-cluster-2014-10.svg