[04:17:51] aude: PruneChanges from WB should probably batch the DELETE query for sanity [04:28:23] aude: and waitForSlaves in EntityPerPageBuilder should use wfWaitForSlaves since $lb->getMaxLag() so too cached [04:30:32] * pastacat notices random stuff [04:38:24] though I guess if getLagTimes gets hit enough it regenerates a lot [14:50:21] 6MediaWiki-API-Team, 10Librarization, 5Patch-For-Review, 7Upstream: Composer autoloader is slow - https://phabricator.wikimedia.org/T85182#1198037 (10JanZerebecki) Enabled it for Wikidata: https://github.com/wmde/WikidataBuildResources/commit/31c9cc34a9372cbb4f95989c5929ccdb4e6123f4 Build for beta with it:... [15:02:42] w00t. New Logstash Elasticsearch servers finally got purchase approval! [15:03:01] * bd808 needs to get serious about fixing up the needed config [15:04:10] <^d> \o/ [15:05:48] <^d> bd808: Shouldn't all those logstash::conf calls go in the actual module? [15:06:15] * ^d sees messy roles :p [15:06:32] I don't think so. Think of them as apache vhost configs vs the apache core module [15:06:44] the role sets things up as we want it [15:07:19] but there could easily be another logstash backend use case that needs different processing setup [15:07:49] hiera could probably be used to clean some stuff up if we wanted to [15:07:56] <^d> Was just about to say [15:08:54] I seem to have a rapidly growing list of "extra" projects to work on lately [15:09:20] and of course most of them are being driven by external deadlines :/ [15:18:15] * bd808 thanks legoktm and Keegan|Away for their hard work on SUL [15:27:33] +∞ [15:43:38] ^d: https://gerrit.wikimedia.org/r/#/c/203284/ [15:44:01] :) always a new nick [16:21:49] <^d> pastacat: https://gerrit.wikimedia.org/r/#/c/203361/ is going to end up with a comment telling you to submit a PR on github [16:22:17] weee [16:22:36] * pastacat hates that whole repo [16:23:23] legoktm: Is there something I can read that explains why the GlobalUserMerge SUL script has to run all at once? I would have thought that it could stop at any time for any length of time, but that's not what Keegan said in an email. [16:43:38] Do we keep metrics on MediaWiki tarball downloads anywhere? [16:44:55] apache access logs? [16:45:39] *nod* So I'd need to talk to analytics? [16:45:59] (aside) there should be a dashboard for this [16:47:17] I hear analytics is all about dashboards [17:14:58] <^d> Heh, we have 3 extensions calling a non-existent core function [17:15:16] <^d> evilMediaWikiBootstrap.php! [17:18:44] meeple27: I don't know what Keegan|Away's email said, but we can pause the script whenever [17:20:04] legoktm: "2. The script breaks and cannot be fixed for any length of time longer than a few days, leaving the process in limbo. In this case the renames that have been done need to be undone for continuity; this project has to happen all at once." [17:20:44] I understand we would want to do a single sweep, but would things actually break if paused for "a while"? [17:21:07] meeple27: not that I can think of on the technical side...maybe it's a community thing? [17:21:24] ok. I'm just trying to understand the risks. I'll follow up with Keegan [17:28:15] 6MediaWiki-API-Team, 10MediaWiki-extensions-CentralAuth: CentralAuth user cache for $mBeingRenamed not being properly invalidated post-rename in some cases - https://phabricator.wikimedia.org/T94491#1198478 (10Legoktm) There was another instance of this reported on the global-renamers list a few days ago. [17:41:22] meeple27: That smells like a community thing and not a technical one [17:43:59] 6MediaWiki-API-Team, 10MediaWiki-extensions-CentralAuth, 5Patch-For-Review: CentralAuth user cache for $mBeingRenamed not being properly invalidated post-rename in some cases - https://phabricator.wikimedia.org/T94491#1198623 (10Legoktm) p:5Triage>3Normal a:3Legoktm [17:48:05] pastacat: is https://phabricator.wikimedia.org/T87397 ("Create separate job loop for LocalRenameUserJob") somethign we should do before the mass SULF renames? [17:49:20] legoktm: remind me, what happens to a user while queued for rename? Can they still login or do stuff? [17:49:30] they're locked out of their account [17:50:42] yeah, so a quick puppet change to have a loop seems reasonable [17:52:41] pastacat: ok, can you take care of that? :) [17:53:26] I suppose [17:55:06] legoktm: awesome. I was going to poke you about that because I put "overloading job queues" down as a possible failure mode for RobLa/Erik [17:55:44] bd808: :) do you have any ideas on what's up with https://phabricator.wikimedia.org/T89169#1189640 ? most of fatal.log has useless backtraces like that [17:56:05] 6MediaWiki-API-Team, 6Availability-Team, 10MediaWiki-JobQueue, 10SUL-Finalization: Create separate job loop for LocalRenameUserJob - https://phabricator.wikimedia.org/T87397#1198709 (10Legoktm) p:5Normal>3High a:3aaron [17:56:16] hmmm [17:57:04] I think that means that the handler is called during teardown following the error after the stack has already been unwound. :/ [17:57:36] which kind of makes sense when you see how the handleFatalError() function is installed [17:58:29] 6MediaWiki-API-Team, 10MediaWiki-Debug-Logging: Fatal Error: Class undefined: MediaWiki\Logger\DateTimeZone - https://phabricator.wikimedia.org/T95727#1198732 (10Legoktm) 3NEW a:3Legoktm [17:58:44] So hhvm is being nice and sending us a signal that PHP5 would not, but sending it at a point where it really doesn't give us any useful information [17:59:37] We may have to build an hhvm version of the wmerrors extension after all [17:59:43] :/ [18:00:12] what we have right now is still better than nothing becase we're getting the url [18:00:42] yeah that's better than nothing [18:01:02] meeple27: I have no idea what I was saying, I'll correct that [18:01:49] 6MediaWiki-API-Team, 10MediaWiki-Debug-Logging, 5MW-1.25-release, 5Patch-For-Review: Fatal Error: Class undefined: MediaWiki\Logger\DateTimeZone - https://phabricator.wikimedia.org/T95727#1198755 (10Legoktm) [18:04:04] 6MediaWiki-API-Team, 10MediaWiki-Debug-Logging, 5MW-1.25-release, 5Patch-For-Review: Fatal Error: Class undefined: MediaWiki\Logger\DateTimeZone - https://phabricator.wikimedia.org/T95727#1198763 (10bd808) p:5High>3Unbreak! [18:04:42] bd808: if that's unbreak, should we be backporting and deploying that now? [18:04:54] I'm clicking buttons in gerrit [18:06:14] 6MediaWiki-API-Team, 10MediaWiki-extensions-CentralAuth, 5Patch-For-Review: CentralAuth user cache for $mBeingRenamed not being properly invalidated post-rename in some cases - https://phabricator.wikimedia.org/T94491#1198792 (10Legoktm) 5Open>3Resolved [18:12:02] I just need to whine... jenkins is slooooow [18:12:31] Our "unit tests" need some tlc [18:12:49] https://gerrit.wikimedia.org/r/#/c/202469/ [18:13:12] oh, that was abandoned [18:13:24] "use PHPUnit_Framework_TestCase directly" +2 [18:22:50] 14 minutes and counting for https://gerrit.wikimedia.org/r/#/c/203376/ to merge [18:24:11] Keegan: Cool, thx. [18:24:38] 6MediaWiki-API-Team, 10SUL-Finalization, 10Wikimedia-Site-requests: Set $wgCentralAuthCheckSULMigration = true in production - https://phabricator.wikimedia.org/T95735#1198983 (10Legoktm) 3NEW [18:25:50] 6MediaWiki-API-Team, 10SUL-Finalization, 10Wikimedia-Site-requests: Set $wgCentralAuthCheckSULMigration = true in production - https://phabricator.wikimedia.org/T95735#1198997 (10Legoktm) [18:29:05] bd808: we can do ^ whenever right? [18:29:51] legoktm: yeah [18:38:10] 6MediaWiki-API-Team, 10MediaWiki-Debug-Logging, 5MW-1.25-release, 5Patch-For-Review: Fatal Error: Class undefined: MediaWiki\Logger\DateTimeZone - https://phabricator.wikimedia.org/T95727#1199074 (10Legoktm) 5Open>3Resolved Deployed by @bd808 [18:48:24] 10MediaWiki-Core-Team, 6Multimedia, 6Parsoid-Team, 6Release-Engineering, and 3 others: Prepare Platform/Ops April 2015 quarterly review presentation - https://phabricator.wikimedia.org/T91803#1199132 (10Qgil) [18:50:32] 6MediaWiki-API-Team, 10MediaWiki-Debug-Logging, 6Release-Engineering, 10Wikimedia-Logstash, and 2 others: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1199157 (10bd808) >>! In T89169#1189640, @Legoktm wrote: > Progress! We now have logs that l... [19:01:46] <^d> Hrm? [19:01:47] <^d> 132 error: String length exceeded 2^31-2: 2147483647 in /srv/mediawiki/php-1.25wmf24/includes/Title.php on line 264 [19:05:23] <^d> wfDebugLog( 'AdHocDebug', json_encode( debug_backtrace() ) ); [19:05:36] * ^d sighs [19:06:57] we've got a stacktrace 2GB long? :P [19:07:04] <^d> Krenair: Your fault :( [19:07:39] <^d> Currently ~130 / last 1000 log entries [19:13:27] :( [19:13:45] wait, 2gb? [19:14:25] whaaaat [19:15:00] Krinkle, ^ [19:15:46] Should we switch to wfDebugBacktrace or wfGetAllCallers? [19:16:46] not sure if get all callers returns everything useful to debug those [19:18:05] Ha! [19:18:06] <^d> I wish we had a way to log what queries triggered an OOM [19:18:19] <^d> Although I suppose stacktraces will help again [19:18:35] ^d: Can we determine what error came before that? Then we can just fix the error instead of the error from the error. [19:19:21] But yeah, wfDebugBacktrace() sounds like a good idea [19:19:21] <^d> iono [19:19:44] <^d> Does it have to be json-ized? It makes for very illegible log files [19:20:07] ^d: String is fine, too. [19:20:13] It was just a quick hack since we can't log arrays [19:20:21] <^d> var_export :) [19:21:15] Krenair: I'll patch? [19:21:45] yes please [19:22:46] <^d> Thanks guys [19:23:23] ^d: I'm still confused. What is being exceeded. [19:23:31] ^d: We don't limit stack traces in other places. [19:23:53] exceeding wfDebugLog? or json_encode? [19:24:58] <^d> I imagine the latter [19:26:34] <^d> See also https://phabricator.wikimedia.org/T70423? [19:27:03] <^d> Also https://github.com/facebook/hhvm/issues/3991 for a similar error msg [19:51:45] 10MediaWiki-Core-Team, 10Librarization, 10MediaWiki-Debug-Logging, 10MediaWiki-Installer, 5MW-1.25-release: Missing log library when trying to install MediaWiki via WebInstaller (mw-config/index.php) - https://phabricator.wikimedia.org/T88951#1199455 (10demon) >>! In T88951#1024053, @bd808 wrote: > `mw-c... [19:57:19] aude: FlaggedRevs doesn't trigger triggerOpportunisticLinksUpdate, not sure if that effects the wikidata usage of that hook... [19:57:41] I guess it might miss some languages in some cases [20:08:54] pastacat: not sure how important that is, but still should look into it [20:09:04] and make sure daniel is aware [20:09:40] whenever I go to fix something I notice like 3 other things [20:10:26] yeah :/ [21:47:28] 6MediaWiki-API-Team, 10MediaWiki-extensions-OAuth, 5Patch-For-Review: Add way for OAuth apps to only authenticate (no other valid rights) - https://phabricator.wikimedia.org/T88757#1199880 (10csteipp) >>! In T88757#1019929, @Mitar wrote: > Hm, why only permissionless apps? Other big sites which use OAuth do... [22:03:24] 6MediaWiki-API-Team, 10MediaWiki-extensions-CentralAuth, 10SUL-Finalization: Code review CentralAuth's GlobalUserMerge and related UserMerge code before SUL finalization mass usage - https://phabricator.wikimedia.org/T961#1199910 (10Legoktm) @brion, hows the review going so far? [23:16:49] legoktm: how swamped are you atm? [23:26:06] 6MediaWiki-API-Team, 10MediaWiki-extensions-OAuth, 5Patch-For-Review: Add way for OAuth apps to only authenticate (no other valid rights) - https://phabricator.wikimedia.org/T88757#1200120 (10Mitar) Security is always a trade-off. Usability, security, attention, habituation, all those things. The issue is th... [23:44:59] * YuviPanda adds more RAM to legoktm