[16:09:42] "There seems to be a problem with your login session; this action has been cancelled as a precaution against session hijacking. Go back to the previous page, reload that page and then try again." [16:09:44] * Reedy kicks his dev wiki [16:11:15] looks like restarting memcached is enough [16:12:13] or not [16:12:14] ( ! ) Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/wiki/mediawiki/core/includes/libs/objectcache/MemcachedBagOStuff.php on line 79 [16:12:44] Bleugh, keep being logged out [16:31:53] timeout on trying to change password [16:44:37] [session] SessionBackend "1pqe3sul87bq0ro5jf2n41on1hcpdjpf" data dirty due to addData(): call_user_func_array/MediaWiki\Session\SessionBackend->MediaWiki\Session\{closure}/MediaWiki\Session\SessionBackend->save/MediaWiki\Session\CookieSessionProvider->persistSession/MediaWiki\Session\SessionBackend->addData [16:47:11] Seem to be stuck in a login loop [17:02:59] anomie: Around? Any chance you could help me sort out sessions on my devwiki? [17:03:18] I dunno if one of the php security updates has upset it [17:05:01] AFAIK... only sessions/staying logged in seems to be broken [17:07:42] Reedy: I can try. First thing, near the start of the log for each request there should be a line telling you what caches are being used for everything. [17:07:53] e.g. "caches INFO: cluster: RedisBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: RedisBagOStuff, parser: RedisBagOStuff, session: SqlBagOStuff" [17:08:02] you mean, incase actual doesn't match LS? [17:08:35] Mainly to make sure the session cache isn't something stupid. [17:08:52] it should be memcached-pecl etc [17:09:54] [objectcache] MemcachedPeclBagOStuff::__construct: persistent Memcached object already loaded. [17:09:59] I see a lot of SessionManager dirty stuff [17:10:03] * Reedy looks for caches INFO [17:11:01] no INFO appearing on mine... but [17:11:01] [caches] cluster: MemcachedPeclBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: EmptyBagOStuff, parser: MemcachedPeclBagOStuff, session: MemcachedPeclBagOStuff [17:11:02] [caches] LocalisationCache: using store LCStoreCDB [17:12:31] https://p.defau.lt/?lov0aaXsSsX3hCF1SW1xPg [17:13:53] ah, hang on [17:13:54] [objectcache] Memcached error for key "WANCache:v:wikidb-mw_:messages:en-gb:hash:v1" on server "127.0.0.1:11211": ERROR was returned by server [17:13:55] Reedy: Let's sanity-check memcached quick, can you telnet into it, execute the "stats" command, and make sure the value for 'time' is current? [17:14:15] let's see [17:14:17] reedy@ubuntu64-web-esxi:/var/www/wiki/mediawiki/core$ php maintenance/mctest.php [17:14:17] 127.0.0.1:11211 set: 100 incr: 100 get: 100 time: 0.099098920822144 [17:14:19] * anomie has had problems with memcached time drift [17:15:02] STAT time 1478538889 [17:16:01] ok, timestamp is good. Are you seeing memcached errors like you quoted above for keys starting with 'MWSession:'? [17:16:02] 1478538956 [17:16:49] Numerous are showing NOT FOUND [17:18:17] Reedy: Next thing to check would probably be to look at your HTTP requests and responses, and make sure the browser is sending cookies back matching the Set-Cookie headers. [17:19:03] [session] SessionBackend "f2bha4574qoprundm5v30iotl9hqplop" data dirty due to dirty(): LoginSignupSpecialPage->getFakeTemplate/SpecialUserLogin->getToken/MediaWiki\Session\Session->getToken/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty [17:19:04] [session] SessionBackend "f2bha4574qoprundm5v30iotl9hqplop" save: dataDirty=1 metaDirty=0 forcePersist=0 [17:20:38] interesting.. opening it in FF [17:20:39] [error] [3974003f90313448af3ee11d] /wiki/Main_Page ErrorException from line 87 of /var/www/wiki/mediawiki/core/includes/libs/time/ConvertibleTimestamp.php: PHP Warning: preg_match() expects parameter 2 to be string, array given [17:20:39] #0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array) [17:20:39] #1 /var/www/wiki/mediawiki/core/includes/libs/time/ConvertibleTimestamp.php(87): preg_match(string, array, array) [17:23:14] There's a lot of those... [17:25:28] Hmm [17:25:34] It seems to be caching a lot of messages into memcached [17:25:39] I wonder if that's doing something like evicting sessions [17:25:41] ConvertibleTimestamp.php errors aren't anything to do with the session stuff [17:26:03] they seem to be transient [17:26:10] /$wgMessageCacheType = 'memcached-pecl'; # optional [17:26:10] $wgMessageCacheType = CACHE_NONE; [17:26:19] Log entries about the session being dirty is ok, that's just logging that it'll have to actually save later. [17:26:42] [objectcache] get(WANCache:v:wikidb-mw_:MessageBlobStore:pdfhandler.messages:en-gb:d136a77bb09344fc84647f6256845003) [17:26:42] [objectcache] result: NOT FOUND [17:26:42] [objectcache] add(WANCache:v:wikidb-mw_:MessageBlobStore:pdfhandler.messages:en-gb:d136a77bb09344fc84647f6256845003) [17:30:00] /var/www/wiki/mediawiki/core/includes/libs/time/ConvertibleTimestamp.php:82: [17:30:00] array (size=4) [17:30:00] 0 => int 1 [17:30:00] 1 => string 'wikitext' (length=8) [17:30:00] 2 => int 2592000 [17:30:01] 3 => float 1478539661.6747 [17:32:27] Every other request it's doing that it seems [17:32:44] from viewaction checkLastModified [17:36:00] can somebody +2 this so I can fix part of a prod ELK logging problem? -- https://gerrit.wikimedia.org/r/#/c/320231 [17:37:27] looks lik erik just did [17:37:47] he did :) [17:38:01] getLastPurgeTimestamp is returning an array [17:38:09] * @return string|bool TS_MW timestamp or false [17:38:13] return $cache->get( $cache->makeKey( 'page', 'last-dc-purge', $this->getId() ) ); [17:40:07] https://github.com/wikimedia/mediawiki/commit/c84ba4d86420d7af918e572e2cd4613d7be185b3 [17:40:14] Seems AaronSchulz changed this on 11th September [17:42:55] * Reedy tags this 1.28 [20:30:12] Hm.. strange. My vagrant won't boot anymore. Haven't updated mediawiki-vagrant in a while just suspended the VM and trying to resume it. [20:30:23] Trying vagrant reload shows: [20:30:24] E: Archives directory /vagrant/cache/apt/partial is missing. - Acquire (2: No such file or directory) [20:30:38] vagrant provision shows the same error + '==> default: Error: Could not run: Could not find file /vagrant/puppet/manifests/site.pp' [20:31:15] bah. that happens when the stupid vbguest plugin tries to run at VM start [20:31:34] or when mounts are broken for some other reason [20:32:04] sounds more generally like the mounts are not working [20:33:40] bd808: Aye, not sure what to do about it though? [20:34:20] Krinkle: I'd try `vagrant halt; vagrant up` first [20:34:49] I've never really used suspend/resume [20:35:21] I did a reload [20:35:34] I'll try halt/up [20:35:46] Looks like it does the same though. [20:35:56] bd808: It's suspended whenever I close my macbook [20:36:02] Been doing that for over a year [20:36:44] Yeah, fails at the same point as reload. https://gist.github.com/Krinkle/3a9738d66238f8d5a0cf1cc8fdf8bdca#file-gistfile1-txt-L51 [20:37:10] ok. that's the vbguest plugin (that I hate) [20:37:23] I'll paste a fix/workaround [20:38:00] Krinkle: https://phabricator.wikimedia.org/P2955 [20:38:43] If you put that in your mw-vagrant root as sibling of the Vagrantfile it will turn off the vbguest plugin [20:41:25] bd808: Worked :) Thanks [20:41:28] bd808: Anything else I should do? [20:41:46] Seems like there's been a major version bump in VirtualBox. Switch from 4.3 release to 5.1 [20:41:57] Is it safe for me to just upgrade VirtualBox separately? [20:42:11] or would that break thigns? Curious how the guest plugin got to version 5.1 or did mwv do that? [20:42:23] yeah it should be fine to upgrade [20:43:46] Needed https://gerrit.wikimedia.org/r/#/c/316416/ as well for provision to pass :) [20:43:51] All good now. Yay [20:45:28] Krinkle: hving mismatched vbguest extenstions in the VM is usually fine. I think I'm going submit a patch to rip the autoupdate extension out of our automatic setup [20:45:38] it causes more problems than it solves [20:45:51] bd808: What does it solve :) [20:46:07] exactly ;) [20:46:47] Its nice on the first boot of a clean vm. once our puppet code has run then its going to break [22:22:52] Krinkle, I encounter that problem when I install VirtualBox upgrades. My workaround is to SSH in to the VM, and mkdir -p /vagrant/cache/apt/partial [23:18:46] #til you can nest functions in PHP [23:18:46] ie: function foo() { function bar(); } [23:18:46] Is totally legit [23:18:46] But! bar() isn't declared until the first time you call foo() [23:18:46] And even then, it remains globally namespaced, not local to the function's scope [23:18:46] How bizarre and useless. [23:56:25] tgr: I've updated PageViewInfo ACL to grant the relevant +2 and Submit permissions to Jenkins and l10nbot [23:56:42] (was missing) [23:56:45] thanks! [23:56:56] tgr: I can make the old repo read-only if you want. [23:57:32] Those should inherit. [23:57:44] (an extension, usually, needs no ACL other than owner) [23:57:48] I want to empty it out first (after moving the pending patches) [23:57:56] ostriches: exclusive? [23:58:16] (the strict version so that "Public and Submit" doesn't show up for maintainers) [23:58:16] so that it doesn't come up when people are grepping mediawiki/extensions [23:58:25] tgr: k [23:58:27] JenkinsBot, Project Owners, l10n-bot, mediawiki all have Submit on 'mediawiki' [23:59:00] And CR+2 [23:59:10] Er, V+2 for JenkinsBot [23:59:12] ostriches: most deployed and "maintained" extensions at this point have exclusive set on V and Submit to that Jenkins cannot be bypassed. [23:59:27] mw-core is one of the few that doesn't do this yet. [23:59:53] (it's often bypassed by accident as well when the previous patch's V+2 is there and you can press "Submit" etc.)