[00:26:59] tgr, anomie: the need_token graph is not getting any better. [00:27:27] it appears to continue to grow linearly as time progresses [00:54:29] https://github.com/pear/Mail/pull/4/files [00:54:32] "Make code php 5 only" [00:54:33] trolol [01:01:38] close, decline [01:06:11] it's dropping php4 shit [01:07:00] precisely, WTF? people still use 4! [01:22:06] Reedy: haha. almost all of PEAR is PHP4 compatible. A whole lot of GCI tasks to clean that up :) [01:25:44] scary [01:30:55] I have a sliiiight suspiion that only PHP4 uses are using PEAR these days anyway... [01:35:58] greg-g: Good thing I didn't actually update my bot yet to use the new OAuth stuff from wmf.11... :( [01:41:28] the need_token graph has legitimately dropped now [01:41:55] I suspected that those bots went into some self-perpetuating error state but that's apparently not the case [01:42:29] in which case, I have no idea what was going on with them [01:42:55] guess need digging into which bots it actually is... [01:43:32] anomie: oh shit. will we have folks broken by that? [01:43:38] after https://gerrit.wikimedia.org/r/#/c/265871/ was deployed I didn't see any obvious difference from the old cookie headers [01:43:43] was batopassword in 10? [01:43:48] *botpassword [01:44:23] what's the new oauth stuff? [01:44:31] owner-only was in 10 I think [01:45:05] there are patches in .11 that grant more rights that anomiebot needed [01:45:19] basically things that were just missed before [01:46:47] tgr: the truly baffling thing about the need_token graph is that the rate continued to climb [01:47:14] bd808: bot passwords are wmf11, yes [01:47:39] so anybody who has switched to a botpassword will now be unable to login? [01:55:06] bd808: yes [01:55:30] plane is leaving but we should write an incident report I think? [01:56:18] yeah, we should [01:56:28] fly safe [03:05:54] legoktm: sorry for late response...but yeah, re-list and push again, that's what I do [03:35:33] bd808: re: the graph climbing, one possible explanation is that there are many affected clients, and as more and more hit their 30-day login expiration, more and more got into the need_token loop [03:35:46] or even just a few clients but they worked on many wikis [16:45:02] tgr: we were running a script to force logout accounts at the same time too right? That would have likely have been causing a number of new login attempts [17:35:17] bd808: https://grafana.wikimedia.org/dashboard/db/authentication-metrics?from=1453010400000&to=1453570478549&var-entrypoint=api&panelId=10&fullscreen [17:35:31] forced logouts started at 1/22 18:15 [17:35:39] there is an effect, but not a huge one [17:36:30] yeah, a bit of a jump but nothing like the needs_token graph [18:42:43] bd808, tgr: the logout script is still running... [18:43:16] legoktm: I don't think it will hurt anything but you could kill it for now I guess [18:43:24] also wow that's slow [18:43:41] (or we have a ton of live sessions I guess) [18:44:08] it does one db write for every user individually [18:44:25] ah. yeah that's never going to be quick