[05:51:37] TimStarling: the phplint55 on ops/mw-config is save to remove, right? [05:51:44] I mean, we can't be running php5 anymore. [05:51:58] Although I do notice we don't have a php7 and hhvm linter. [05:52:05] So it's not as straight forward [05:52:12] well, we do have them, but not on that repo. [05:56:32] Uploaded https://gerrit.wikimedia.org/r/#/c/integration/config/+/479158/ - don't have JJB ready at the moment so will deploy tomorrow. [05:57:05] but if you want, can try via https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Deploy_changes [05:57:13] * Krinkle checks out [05:57:14] o/ [06:48:51] <_joe_> Krinkle: absolutely safe in theory, yes [10:57:38] RoanKattouw: Hey, this page has a template saying it has outdated information: https://wikitech.wikimedia.org/wiki/How_to_do_a_schema_change Is it because you don't want to slap people in the face anymore? I can help if you're tired [18:00:55] James_F: so, the good news is: I was wrong, there is no conceptual problem blocking MediaInfo deployment. [18:01:04] the bad new is: I was wrong and I have no idea what the problem is. [18:01:06] https://phabricator.wikimedia.org/T204748 [18:01:13] duesen: :-( [18:02:10] The behavior is consistent with onWikibaseRepoEntityNamespaces not running. The behavior is however not consistent with wgMediaInfoEnable being false, which would also disable other hooks [18:02:20] Yeah. [18:02:39] Does the value get over-written because of the lack of a prefix or something? [18:02:44] *something* is going wrong when loading entities. but I can't guess what. [18:02:55] which value? [18:03:09] Sorry, the values inserted by the WikibaseRepoEntityNamespaces hook. [18:04:24] I don't see a chance for this to happen. I though that this kind of thing may happen when the config arrays get merged. But they don't get merged. Wikibase operates in "split brain" mode, using one config for the client code, and another config for the repo code. Which is conceptually really bad, but in practice means this actually works :) [18:04:29] it does work for me locally. [18:04:40] Yeah. [18:04:56] Is T211801 something the Wikibase team can do, do you think/ [18:04:56] T211801: Find out how to deploy MediaInfo without breaking Commons - https://phabricator.wikimedia.org/T211801 [18:05:05] I put my findings into two new tickets, please have a look and let me know if you have further questions [18:05:59] I can't answer that question, you'd have to ask leszek and lydia. my guess is that they don't have resources to sink into this [18:06:29] I could probably spend another half day or so investigating. but since i was unable to reproduce locally, I'd need to find out how to debug on beta itself [18:06:35] I have no clue how to do that [18:07:06] also, we'd have to break beta again i guess [18:07:19] (wouldn't it be nice if we could just spin up another instance iwth the broken config?...) [18:09:48] Right. [18:23:18] Amir1: The basics should still be up to date (foreachwiki sql.php) [18:23:30] Amir1: I also don't have to fly to SFO any more to do that, I live here now :) [18:23:55] RoanKattouw: good point :D [18:25:44] RoanKattouw: Says the guy flying from SJC. ;-P [18:30:29] I hope you don't need to fly to TXL, that's going to be a reallly long flight [18:30:35] RoanKattouw: Can you tell me how I would debug something live on the beta cluster? I'm not sure where to start. [18:31:26] logstash-beta.wmflabs.org sometimes help duesen + mwscript eval.php on deployment-deploy01.eqiad.wmflabs [18:34:14] If you need to change the code, you can change it on deployment-deploy01 (but tread lightly) [18:34:20] Otherwise, what Amir said [18:34:41] Amir1: that's better than nothing, I suppose, thanks. Where do I find the credentials for logstash-beta.wmflabs.org? The suggestion to ssh deployment-deploy01.deployment-prep.eqiad.wmflabs does not work. [18:35:11] Doesn't deployment-deploy01 get auto-synced to "real" Beta Cluster? That feels like a bad outcome in terms of system stability. [18:35:15] that's where they're stored, if you are a member you can view it [18:36:06] hrrrm. I suppose my ssh config is still not right for this [18:36:13] I keep having trouble with that. [18:36:42] what's your ldap username? [18:36:45] duesen: ^ [18:36:51] duesen: isn't it in Logstash (ssh deployment-deploy01.deployment-prep.eqiad.wmflabs sudo cat /root/secrets.txt? [18:37:03] Last time I got it from there [18:37:58] I don't think he's a member of the deployment-prep project, hence me asking for his ldap name :) [18:38:22] > Connection closed by UNKNOWN port 65535 [18:38:26] o_O [18:38:34] port 65535? [18:39:36] it's a valid port, so why not? [18:39:50] it's highly suspiciouos [18:39:56] something is looping and running out of ports [18:40:17] the point being: this is the *last* valid port. [18:41:27] I before ran into an infinite loop with the proxy setup. the error was a bit different though. [18:41:30] *sigh* [18:41:41] * duesen is not a good server admin [18:41:42] ah, I seemed to have misunderstood the context. I thought that was the src port of something trying to connect to you, rather than ports you're using. Does netstat show literally every port in use? Might be able to find the pids that way to see what's looping through all them [18:42:27] i know what's looping. ssh. i'm trying to connect to deployment-deploy01.deployment-prep.eqiad.wmflabs via a proxy config, and i'm failing [18:44:00] greg-g, Amir1 any idea what I'm doing wrong? https://pastebin.com/7vy0TmvT [18:44:20] is this correct? ---> ProxyCommand ssh -a -W %h:%p daniel@primary.bastion.wmflabs.org [18:46:38] duesen: for me it's ProxyCommand ssh -a -W %h:%p ladsgroup@bastion.wmflabs.org [18:46:52] try without "primary." [18:47:11] just for the record, can we get back to my question, I don't see a "daniel" in the deployment-prep project. [18:48:01] duesen: try again [18:48:06] I just added you to the project. [18:48:47] greg-g: heh, that did it, thank you :) [18:49:21] the error message is a bit unhelpful... [18:49:28] welcome to ssh [18:52:47] RoanKattouw: This is the last bit of deprecating tag_summary https://gerrit.wikimedia.org/r/c/mediawiki/core/+/476069 [18:53:06] (before stopping to update the table and dropping it) [20:12:50] James_F: (daniel really but he's not here) as a last-ditch effort you can use the HHVM debugger: https://www.mediawiki.org/wiki/Manual:How_to_debug#HHVM [20:13:08] I haven't tried doing that in production but Erik said it works well [20:13:22] Huh. Never tried that. [20:13:29] I'll see how desperate we get. [20:13:32] I guess you could set up XDebug too but that's more scary security-wise [20:13:53] (also cool people use shell.php not eval.php!) [20:14:02] Yeah yeah. ;-) [21:12:54] <_joe_> anomie: what would the keys for the sessions look like on redis? [21:14:57] _joe_: On enwiki, prefix is "enwiki:MWSession:" followed by a 32-character session ID string. Swap the "enwiki" for other wiki's wikiids as appropriate. [21:15:04] <_joe_> I see a lot of data for echo being stored on those servers, like "global::echo::seen:..." [21:15:10] <_joe_> anomie: ack, what I thought [21:15:34] <_joe_> now, my issue is: do we need the same service to work for echo too [21:16:04] <_joe_> I'll comment on the ticket [21:16:23] <_joe_> the amount of echo data in those redises is insane, btw [21:29:20] _joe_: more generally, right now whether something goes into memcached or redis is somewhat arbitrary [21:30:04] if we go active/active and memcached won't support multi-dc sync and the successor of redis will then that will probably have to be rethought [21:30:23] I'm pretty sure sessions are not the only thing that needs cross-dc consistency [21:30:24] <_joe_> tgr: memcached works cross/dc [21:30:39] <_joe_> it can be controlled so to be consistent [21:30:59] not well enough for us to use it for sessions, right? [21:31:02] <_joe_> tgr: the distinction is: do you need active/active writes? [21:31:33] <_joe_> tgr: we definitely don't want to have sessions in memcached, it's too volatile [21:31:56] yes, and I don't think anyone reviewed which BagOStuff usages need that [21:32:04] <_joe_> it is both consistent and not guarantee persistence [21:32:34] <_joe_> tgr: I was under the impression that only cached data would go to memcached [21:32:41] <_joe_> so you need consistent *eviction* [21:32:45] <_joe_> not consistent presence [21:32:59] <_joe_> AIUI, the echo data is being /stored/ there [21:33:21] the CentralAuth global session is an obvious example of something that's stored in memcached but can be written anywhere and needs to be consistent [21:33:35] I'm fairly sure there are less obvious ones though [21:33:36] <_joe_> tgr: define consistent [21:33:37] <_joe_> :) [21:34:05] <_joe_> tgr: you mean it should be always guaranteed to persist, or it should not be different in the two dcs? [21:34:14] <_joe_> the latter is guaranteed more or less [21:34:36] I don't think we have any cache that must be guaranteed to persist [21:34:54] <_joe_> ok then for such things memcached is perfect [21:35:11] BagOStuff does not guarantee that in general, and we happen to use RedisBagOStuff for sessions but someone else might not [21:35:25] <_joe_> if what you want is "I can have a value or no value, but if I have a value it should not be stale" [21:35:27] so if there is any persistence requirement in MediaWiki that would be a bug [21:35:43] well, except for the job queues which are not via BagOStuff [21:35:52] I'm not familiar with what Echo does [21:35:57] <_joe_> the jobqueue *needs* persistence [21:36:03] <_joe_> yeah let's hear from people that know [21:36:05] <_joe_> :) [21:41:53] <_joe_> tgr: you know who should I ask to? [21:43:17] _joe_: #wikimedia-collaboration [21:43:26] Roan I guess? [21:44:07] the code uses ObjectCache::getMainStashInstance() [21:44:55] * Stashes should be configured to propagate changes to all data-centers. [21:45:13] * Callers should be prepared for: [21:45:17] * - a) Writes to be slower in non-"primary" (e.g. HTTP GET/HEAD only) DCs [21:45:22] * - b) Reads to be eventually consistent, e.g. for get()/getMulti() [21:45:55] oh yeah it also says the data is not stored elsewhere [21:46:25] so someone should probably review usages of getMainStashInstance [21:46:54] at a glance it is used by Echo, CentralAuth, AbuseFilter, ConfirmEdit at least