[04:50:18] !log commonsarchive upgrading MediaWiki to 1.27 [04:50:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive/SAL, Master [05:00:28] !log commonsarchive deleted commonsarchive-test, and on the remaining instances done sudo apt-get update; sudo apt-get dist-upgrade; sudo apt-get autoremove [05:00:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive/SAL, Master [05:07:28] Krinkle: Quick question on intuition. Is it for tool labs only? Or is there a way I can modify the returnto param when changing lanugage? [06:26:10] 06Labs: Add project-wide shared space for wikidata-query project - https://phabricator.wikimedia.org/T138259#2413704 (10Smalyshev) Is there anything needed to make this happen? [06:34:07] 06Labs: Add project-wide shared space for wikidata-query project - https://phabricator.wikimedia.org/T138259#2413706 (10yuvipanda) A strong rationale :) NFS is a PITA, and we'd like to not make it available unless there's a strong reason to. [06:42:24] 06Labs: Add project-wide shared space for wikidata-query project - https://phabricator.wikimedia.org/T138259#2413707 (10Smalyshev) I need a shared space for the project. I tried to use /data/scratch but it was wiped recently so I lost all the files. Nothing critical, but significant inconvenience. I would like t... [07:29:33] !log authmanager rebuilt am-01 (after accidentally deleting it), installed all extensions that have been updated for AuthManager, switched to REL1_27 [07:29:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Authmanager/SAL, Master [07:51:20] (03CR) 10Lokal Profil: [C: 032] Add symlink to Toolbox from HTML page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296056 (https://phabricator.wikimedia.org/T138519) (owner: 10Jean-Frédéric) [07:57:46] (03Merged) 10jenkins-bot: Add symlink to Toolbox from HTML page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296056 (https://phabricator.wikimedia.org/T138519) (owner: 10Jean-Frédéric) [08:53:30] !log commonsarchive downgraded MediaWiki to 1.26 pending fix for T110460 [08:53:31] T110460: Update OAuthAuthentication to use AuthManager - https://phabricator.wikimedia.org/T110460 [08:53:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive/SAL, Master [09:35:36] (03CR) 10Jean-Frédéric: [C: 032] Add .gz to .gitignore [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296468 (owner: 10Lokal Profil) [09:36:24] (03Merged) 10jenkins-bot: Add .gz to .gitignore [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/296468 (owner: 10Lokal Profil) [10:13:13] zhuyifei1999_: is commonsarchive using OAuthAuth? [10:48:12] 06Labs: Can not kill job on tools labs - https://phabricator.wikimedia.org/T138924#2414164 (10Magnus) [10:52:39] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: nlwiki Labs replica with problems - https://phabricator.wikimedia.org/T138927#2414203 (10Magnus) [10:56:32] tgr: yes, it is. [10:58:25] https://gerrit.wikimedia.org/r/#/c/251930/ updates OAuthAuth for 1.27/AuthManager [10:58:45] it's not very polished (hence WIP) but should work [10:58:56] feedback is appreciated etc [11:01:37] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: nlwiki Labs replica with problems - https://phabricator.wikimedia.org/T138927#2414203 (10jcrespo) This is a known issue, the long term reasons are difficult to fix without changing mediawiki iself. The short term ones are being handled with the full (b... [12:05:24] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: nlwiki Labs replica with problems - https://phabricator.wikimedia.org/T138927#2414448 (10Magnus) 05Open>03Resolved a:03Magnus OK, I'll wait, not really mission-critical. [12:54:36] tgr: yes [12:56:53] should I checkout that patch and see if it works perfectly? [14:06:18] zhuyifei1999_: yes, it should work [14:06:32] k [14:06:55] it has too many configuration options for its own good so feedback on which of those are actually useful would be great [14:16:46] !log commonsarchive upgraded to 1.27 & for OAuthAuth (as www-data) git fetch origin refs/changes/30/251930/30; git checkout FETCH_HEAD & added $wgOAuthAuthenticationReplaceLoginLink = true; [14:16:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commonsarchive/SAL, Master [14:20:57] tgr: Exception encountered, of type "BadMethodCallException" [14:21:30] and is there a way to bypass that login form? [14:21:52] see https://commonsarchive.wmflabs.org [14:23:08] there is no way yet to log in without having to press a button on the login form [14:25:26] :( [14:26:05] can you look into the exception? [14:26:06] https://www.irccloud.com/pastebin/MT09RS9t/ [14:26:19] yeah, looking [14:26:29] everything except oauthauth is at REL1_27 [14:26:40] cleared cookies => idem [14:26:42] apparently I did not ever test with the whitelist options enabled [14:27:11] oh [14:29:09] fyi the configuration is [14:29:19] https://www.irccloud.com/pastebin/DwNdRxp4/ [14:31:27] what the faulty piece of code does is check on every request that the user is still a member of the whitelisted group [14:33:00] I'll just disable that for now, I don't think there is a straightforward way to port it to AuthManager [14:33:21] it will still be checked on login, just not on normal requests [14:33:34] k [14:35:10] zhuyifei1999_: try the updated patch [14:37:36] k [14:39:29] 500 hphp_invoke [14:41:31] ah, mediswiki.log has something [14:41:42] *mediawiki.log [14:42:08] PHP Fatal Error: Class undefined: MediaWiki\OAuthClient\ClientConfig [14:42:30] o.O [14:42:39] zhuyifei1999_: the extension uses composer [14:43:09] if you are not installing it via the vagrant role you need to run composer install manually [14:43:31] k [14:50:58] what's the source of the user/pass for http://shinken.wmflabs.org/user/login ? [14:51:01] 06Labs, 10Tool-Labs: enwiki_p DB corruption - https://phabricator.wikimedia.org/T138954#2414879 (10Betacommand) [14:51:16] it previously did not have a login requirement [14:51:30] greg-g: try guest/guest? [14:52:19] Luke081515: tada! [14:52:23] thanks :) [14:52:27] tgr: since I'm always using mediawiki/vendor.git, do I "composer install --no-dev" directly in the extension dir? it's still the same right now [14:52:44] (same exception) [14:52:58] greg-g: btw, I think the guest login credentials are listed at the login interface too? :D [14:54:01] Luke081515: hush you :P [14:54:45] :D [14:55:51] zhuyifei1999_: can you verify that vendor/mediawiki/oauthclient/ exists in the extension dir? [14:56:24] 06Labs, 10Tool-Labs: enwiki_p DB corruption - https://phabricator.wikimedia.org/T138954#2414879 (10jcrespo) I only see 1 now. I would assume this is a very recent change and enwiki will continue to have lag https://tools.wmflabs.org/replag/ due to the reimport that it is in progress for the last weeks. Let's w... [14:56:38] yes [14:56:50] https://www.irccloud.com/pastebin/AXhXR4NE/ [14:57:52] but not in https://commonsarchive.wmflabs.org/wiki/Special:Version [15:00:39] that's normal, I don't see it in S:V in my local install but it works fine [15:00:47] well, not normal, but that's not the problem [15:00:51] hmm [15:01:15] also there should be way more files [15:01:51] can you make me a project member? [15:04:54] k [15:07:04] LDAP user = Gergő Tisza? [15:07:32] yes [15:08:18] done [15:09:10] fyi I pasted the logs at https://phabricator.wikimedia.org/P3312 [15:16:25] the logs don't tell much apart from that this is an autoloading problem [15:19:45] what is jcrespo's IRC nick? [15:20:10] Betacommand: jynus [15:21:38] jynus: what DB are you using to query T138954 ? Im still getting 37 when I just double checked. [15:21:39] T138954: enwiki_p DB corruption - https://phabricator.wikimedia.org/T138954 [15:22:22] Betacommand, I meant production [15:22:38] what I meant is let's wait until lag goes back to 0 [15:22:43] then reevaluate [15:23:48] jynus: its been lagging and wrong for quite some time, far more than the 1-2 hour lag thats showing [15:24:05] lagging is intended, I make it lag [15:24:27] jynus: I know that, the lag has nothing to do with the wrong values [15:24:50] well, the updates to the pagelinks or imagelinks are anynchronous [15:25:11] they do not have anything to do with the actual wiki updates [15:25:25] so I say let's wait and do the test in a few hours [15:25:28] it will not hurt [15:25:56] in any case I cannot fix things when lagged because I am fixing another table at the same time [15:26:03] so it is not a choice :-) [15:26:53] jynus: the changes where made on wiki almost 2 months ago [15:27:27] yes, but it could easily get purged again in the last hour [15:27:33] I am not disputing it being wrong [15:27:41] I am saying, we need to wait [15:27:42] zhuyifei1999_: it seems vendor/autoload.php is not getting loaded [15:28:09] jynus: how long should I wait before re-raising this issue? [15:28:11] which is weird because wfLoadExtension should take care of that [15:28:25] It usually takes 8 hours between reimports [15:28:48] as a horrible hack you can probably require_once it from LocalSettings [15:28:59] it started 6 hours ago [15:29:10] so maybe 2-4 hours [15:30:01] the questin is that I reimported that table a few weeks ago [15:30:12] the fact that it broke again (if it confirms) [15:30:41] means that there is an intrinsinc problem with either the filtering or mediawiki queries [15:31:07] which means I can reimport forever the rows, but never be able to fix it completely [15:31:28] for example, it seems that because deleted revisions are filtered out [15:31:30] tgr: ugh [15:31:34] why not? [15:31:40] thet do not exist on labs [15:31:56] beats me [15:31:59] so if a revision gets un deleted, it doesn't on labs [15:32:11] I don't have access to ww-data so I can't test it [15:32:28] that is a problem of mediawiki or the filtering, depending on how you see it [15:32:38] the code certainly looks like it should load it, extension.json has a truthy load_composer_autoloader key [15:33:05] mediawiki should never do insert * select and move revisions between tables [15:33:49] tgr: sudo [15:34:08] so maybe it is logically impossible to maintain, in the current form, an accurate AND secure labs db at the same time [15:34:17] sudo -- sudo -u www-data -- [15:34:33] (which is kind of like a hack but anyways) [15:34:36] oh ok, I only tried sudoing as www-data [15:34:54] sadly, with a few exceptions, I cannot get mediawiki hackers to consider labs problematic into account [15:35:14] and help is welcome [15:35:39] so, for now, I will continue the reimports :-) [15:35:56] certainly we will arrive to a better state than a few months ago [15:36:48] weird, in CLI mode MediaWiki\OAuthClient\ClientConfig is actually defined [15:38:01] jynus: I'm not sure if people realise the intricacies of the replication setup. Until a while ago, I thought it was just mysql replication with views on top (as it was on the Toolserver), and my current understanding doesn't go much further than 'something something triggers' [15:38:11] https://wikitech.wikimedia.org/wiki/Tool_Labs/Database_plan#Production_Data_Replication is the only piece of documentation I can find on it [15:38:26] which explicitly says it's obsolete... [15:38:37] it goes deeper than that [15:39:07] it goes as far as certain mediawiki queries, under certain circumstances, getting different results [15:39:30] the difference between production and labs is that: [15:39:39] tgr: I think cli 'php' is zend [15:39:41] a) certain tables do not exist [15:39:52] b) it uses a different engine for compression reasons [15:40:03] and the web is hhvm (that's for sure) [15:40:11] c) it suffers different connection patterns [15:40:44] if on top of that we have the fact of multitude of occations where mediawiki generates unsafe write statements [15:40:50] it creates a problem [15:40:58] what do we do in production? [15:41:21] nuclear option- if a server is derected to have a difference, we nuke it [15:41:35] we clone it, which takes 1 hour top [15:41:49] we cannot do that with labs [15:42:06] we cannot copy from production because it has been sanitized [15:42:26] and it cannot be a binary copy because it uses extra compression [15:42:42] which means we have to slowly fix it row by row [15:43:02] how can things get better? [15:43:26] 1) migrate back to InnoDB- that will reduce clone time by 100x? 1000x? [15:43:37] tgr: I got a hhvm cli running using "sudo -- sudo -u www-data bash -c 'LC_ALL=C hhvm /srv/mediawiki/maintenance/update.php --force'" [15:43:39] for that we need space, and for that we need new servers [15:43:49] so my understanding of replicas was 'they repeat every query sent to the master server in order', but that's apparently not true either? [15:44:01] in theory [15:44:04] in practice [15:44:11] let me give you an example [15:44:22] how a revision is undeleted on mediawiki [15:45:06] it is something like "INSERT into revision SELECT * FROM archive WHERE ..." [15:45:36] but it happens that the same querie, executed on 2 servers, creates a different output [15:46:09] in particular, probably that doesn't even exist because deleted revisions are not exposed on labs for privacy reasons [15:46:26] that leads to all kinds of issues [15:46:34] that is the naive explanation [15:46:48] in reality the difference is much more subtle [15:47:03] Ah. So what happens is: A) master moves revision to archive table, B) replica replicates this, C) replica trigger removes that row, D) master moves revision back to revision, E) replica cannot find the row because it has been removed? [15:47:06] like the archive id being different because that id was blocked by another connection [15:47:27] different locks => different ids => slave drift [15:47:38] *nod* [15:47:41] IF the queries were safe [15:47:55] something that is highly recommended in any env [15:47:59] that would not happen [15:48:13] hint: mediawiki is not safe- but I intend to make it so [15:48:41] see my efforts on https://phabricator.wikimedia.org/T108255 [15:48:52] super interesting jynus thanks for explaining [15:48:57] so that is effor #2, migrate to safe statements [15:49:06] and row based replication [15:49:10] in production [15:49:24] which would lead to a better labsdb [15:49:43] effort #3 is reimport all wikis, that may have large issues from a long time ago [15:50:02] that will not fix all issues forever, but that will make them so few [15:50:08] 'safe' in this context means: every query should be fully defined (i.e. no auto increment, no server-side functions, etc) on the php level? [15:50:39] valhallasw`cloud, for the full explation for devels, check https://phabricator.wikimedia.org/T112637 [15:50:53] now, I need volunteers to check extensions [15:51:12] and help propagate the need to use sql_mode=TRADITIONAL [15:51:30] among all developers big and small, employees and volunteers [15:51:57] it is my intention to create a presentation on the next developers conference [15:52:08] and do training session about this [15:52:33] I think it will be impossible to make labs 100% correct, all the time [15:52:49] (many things happen asynchronously, and that is a good thing) [15:53:17] but make the issues so few and spaced that they can be detected and solved fastly [15:53:39] zhuyifei1999_: I ran 'sudo -- sudo -u www-data -- hhvm -a' then replayed the maintenance script initialization [15:53:47] I am about to finish the import of enwiki, other wikis will follow behind [15:53:52] *reimport [15:54:09] at the end, new MediaWiki\OAuthClient\ClientConfig; died with some reasonable error message about the number of constructor arguments [15:54:13] and after that we can evaluate automatic detection and the main offenders [15:54:21] ok [15:54:47] let's try echo commands at random places I guess :) [15:55:30] I think the main offenders are deleting and undeleting pages + auto_increment ids [15:55:35] *nod* [15:55:44] only based on reports [15:56:22] so it's the auto_increment, not the triggers that clean the data? [15:56:29] and before all of that, we need to deploy new, 512GB machines for labsdbs- our most expensive machines ever [15:57:03] so nobody complains ever about them being slow [15:57:37] Heh. The usage will grow to available capacity ;-) [15:57:50] ha, that's true [15:58:10] I may also need your help, chase's, etc to see how to use them properly [15:58:21] jynus: is it OK if I copy this conversation on-wiki? [15:58:35] depending where? [15:59:03] some subpage of Help:Tool_Labs/Database to explain what's happening with replica divergence [15:59:11] sure [15:59:22] I created a comment with some of that in the past [15:59:55] zhuyifei1999_: https://commonsarchive.wmflabs.org/wiki/Special:Version -- autoload.php is not loaded, but only in web mode [16:00:33] legoktm: seen anything like ^? wfLoadExtension does not load the composer autoloader, but only in web mode [16:00:59] there are no permission issues as far as I can tell [16:01:09] I have a kind of an issue with tickets about that [16:01:17] because all of that are right [16:01:27] I cannot close them [16:01:49] and they are useful to track that the efforts work [16:01:56] but they are too many of them [16:02:15] and at some point they keep not being useful, because things are happening [16:02:24] only very slowly for labs users to notice [16:03:15] tgr: should I reboot and see if it fix it? [16:03:37] zhuyifei1999_: worth a try [16:03:49] this is like the typical answer: https://phabricator.wikimedia.org/T134203#2325577 [16:04:19] now that you mention it I ran into a bug in the past where file changes were not picked up by web HHVM, only CLI (or maybe the other way around)? [16:04:29] only restarting the hhvm service solved it [16:04:38] that involved symlinks though [16:05:11] ok rebooted [16:05:41] 68 => '/srv/mediawiki/extensions/OAuthAuthentication/vendor/autoload.php', [16:07:02] hmm that issue is gone, but 'OAuth login error: oauth_callback must be set, and must be set to "oob" (case-sensitive)' [16:07:39] yeah I guess the hhvm cli didn't puck up the changes [16:07:50] *pick [16:13:02] zhuyifei1999_: that sounds like a problem with the OAuth consumer [16:13:28] let me check [16:13:40] k [16:15:23] https://github.com/wikimedia/mediawiki-extensions-OAuth/blob/2512f36bb4162dcf35ee8d253d63dc9fcc8ea216/backend/MWOAuthServer.php#L60 [16:17:11] and yeah, irc the oauth registration was a fixed callback url and client can't supply the calback url [16:17:16] *iirc [16:17:38] yeah, I think the consumer callback is assumed to be dynamic [16:19:01] it must include a CSRF token so oob cannot be used here [16:19:11] should I register another consumer? [16:20:03] the prefix should be https://commonsarchive.wmflabs.org/w/index.php?title=Special:UserLogin/return IIRC [16:20:15] ofc you can't go wrong with just https://commonsarchive.wmflabs.org/ [16:20:25] (but since when csrf is required for oauth?) [16:20:31] yes, unfortunately there is no way to change existing consumers [16:21:18] the auth system has no knowledge of what specific mechanism an auth extension is using so it does not know if it is CSRF-safe [16:21:33] k [16:21:51] there is probably a way to handle things better but I haven't figured it out yet [16:30:01] jynus: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database/Replica_drift [16:30:19] tgr: proposed db44a5648895eb3892012f3331e00004, but now I get this after callback: [16:30:46] https://www.irccloud.com/pastebin/xy4t9ILL/ [16:31:52] actually 3 new servers were ordered [16:32:32] jynus hi, could you restart phab-01.phabricator.eqiad.wmflabs and phab-04.phabricator.eqiad.wmflabs please [16:32:40] since they seem to have a problem restarting [16:32:57] paladox, I do not have rights on labs [16:33:04] jynus oh [16:33:21] I assume those are vms? [16:33:52] jynus yes [16:33:58] my power on labs ends on labs-support network [16:34:06] jynus oh ok [16:34:12] aka dbs and similar stuff [16:34:16] Fault [16:34:17] MessageUnavailable console type serial. [16:39:17] zhuyifei1999_: that seems to be T137182 [16:39:18] T137182: User:Qian.Nivan cannot login at fawiki due to "Unknown or bad timezone" - https://phabricator.wikimedia.org/T137182 [16:42:54] why world +00:00 be invalid [16:43:00] *would [16:43:17] not sure where the date comes fromat all, the session data should only contain a Message [16:48:57] removed the data: and idem [16:49:10] https://www.irccloud.com/pastebin/CxTFkzS6/ [16:51:35] cleared cookie and relogin => idem [16:52:19] jynus: is there a task for the new db servers? [16:52:34] I can't find it, but that's probably more related to my searching skills [16:54:02] there is not, there will be probably a private one for the purchase [16:54:23] but there is not one for the setup because they are not yet here [16:54:26] *nod* [16:54:42] actually [16:54:47] what's the rough ETA? next quarter, 6 months, ? [16:54:53] they are here, it seems [16:55:00] but still in hardware phase [16:55:05] https://phabricator.wikimedia.org/T136860 [16:55:11] so this has to finish [16:55:20] the I have to create another one for the service setup [16:55:31] and that will probably require some downtime [16:56:43] so difficult to say, maybe 3 months? [16:57:08] it depends on if we continue importing and then set them up or we set them up and then import [16:57:24] zhuyifei1999_: some weird default timezone setting maybe? [16:57:32] https://www.irccloud.com/pastebin/Dg89lvl2/ [16:57:51] searching for "+00:00" in logs, got ^ kind of stuffs [16:59:32] jynus: right. I'm writing an e-mail to labs-l about the replica drift (which hopefully helps to reducte the number of duplicate reports). I think I'll just go for 'We hope to have these servers up and running in a few months'. [17:00:19] default timezone? can't find it in LocalSettings.php [17:00:33] the problem is that if I say "next month" (that could be) then if things get delayed [17:00:40] people can complain [17:01:11] sometimes bad things happen, even in in unrelated tasks [17:01:34] *nod* [17:02:14] the question is that I actually do not want less reports [17:02:36] I just would like them in a way that are easy to check and fix [17:02:51] maybe a wiki page or a table or something [17:03:19] https://wikitech.wikimedia.org/w/index.php?title=Help_talk:Tool_Labs/Database/Replica_drift maybe? [17:03:19] so that when the reimport finishes, I can go back to them and say- solved, solved, not solver- to fix [17:03:33] or tasks that are a subtask of some other task? [17:03:51] probably that is unmaintenable [17:04:04] because we could end up with one task per row [17:04:40] and I expect most of them fixed after the reimport [17:04:54] maybe a single task with a list of queries [17:05:15] and expected results vs. found results [17:05:18] zhuyifei1999_: identifyTS only ever gets set to new MWTimestamp() or wfTimestamp() [17:05:23] in the smallest format possible [17:05:40] count(*) or md5sum [17:05:55] of course the expected results will also change with time... [17:06:00] that way I do not miss any, [17:06:03] well [17:06:22] also the value found [17:06:30] and MWTimestamp always sets new DateTimeZone( 'GMT' ) [17:06:33] but I would check them by comparing them to production [17:06:40] so this is some weird PHP problem [17:06:47] sometimes I get false positives [17:06:50] very often [17:07:10] ah, right. That's a good check. [17:07:13] because some page has to be reparsed and the jobs rerun to create new pagelinks, or templatelinks [17:07:17] let me check the session [17:07:29] but in reality the database results are the same than in production [17:09:13] in the future that will be automatized [17:09:22] but again, that has still some issues [17:09:43] because tables in production lack primary keys [17:09:59] https://phabricator.wikimedia.org/T17441 [17:10:09] another crusade of mine [17:10:33] or are so large that is is not sometimes feasable to compare them, like the revision table of enwiki [17:15:11] (03Abandoned) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [17:15:36] tgr: https://phabricator.wikimedia.org/P3316 the session. I don't see it has anything related to timezone [17:17:32] with this session GET https://commonsarchive.wmflabs.org/w/index.php?title=Special:UserLogin&returnto=Main+Page&returntoquery=title%3DMain_Page breaks [17:22:18] * valhallasw`cloud prods wikibugs [17:22:24] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10valhallasw) [17:22:48] jynus: ^ [17:23:17] zhuyifei1999_: it's encrypted [17:23:23] what's the session id? [17:24:07] I added you to the "can view" [17:24:27] the id is right after "commonsarchivewiki:MWSession:" [17:24:32] ah, sorry, didn't notice [17:30:27] thankyou for all of that, valhallasw`cloud [17:30:31] much appreciated [17:31:39] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Prnk28 was created, changed by Prnk28 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Prnk28 edit summary: Created page with "{{Tools Access Request |Justification=I'm currently working on a GSoC project called 'Accuracy Review of Wikipedia' (https://phabricator.wikimedia.org/T129536) under mentor Ja..." [17:32:04] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415464 (10jcrespo) [17:32:56] how might i reset my 2 factor on wikitech? My old phone screen died so i got a new one, but now i need to resetup authenticator [17:33:49] 06Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#2415416 (10jcrespo) [17:33:52] 06Labs, 10Wikimedia-Labs-General, 10DBA, 06Operations, 07Tracking: Database replication services (tracking) - https://phabricator.wikimedia.org/T50930#2415469 (10jcrespo) [17:34:25] ebernhardson: https://wikitech.wikimedia.org/wiki/Password_reset [17:37:27] valhallasw`cloud: i can do a video chat, but you probably don't recognize me :) i'll poke chase or yuvi when around i suppose [17:37:38] ebernhardson: I also don't have access to the database ;-) [17:38:02] ahh, actually i might have access to that db... [17:39:15] indeed i do, done. thanks! [17:39:35] tgr: can I git checkout -- includes/specials/SpecialVersion.php ? [17:41:57] * ebernhardson now needs to figure out the same for phabricator... [17:44:28] ebernhardson: https://wikitech.wikimedia.org/wiki/Phabricator#Removing_Two_Factor_Authentication :-) [17:50:06] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2415513 (10demon) I've done the rename of "Luis Felipe Schenone" to "Felipe Schenone" per the request. Sorry it took so long, I got c... [18:03:17] 06Labs, 10Phlogiston (Interrupt), 15User-bd808: Phlogiston-1 unresponsive - https://phabricator.wikimedia.org/T137736#2377044 (10chasemp) thinking this is related to T137857 I tried to recovery this VM but so far no dice. ``` nova reset-state --active ab9a7a7b-709f-4bb6-9f33-c56942f30ab5 nova start ab9a7a7... [18:03:37] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2415541 (10lfschenone) Thanks a lot demon! About lfs to lfschenone, do you mean we should wait for someone from labs. Maybe when the t... [18:07:51] 06Labs, 06Operations, 06Release-Engineering-Team, 10wikitech.wikimedia.org: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2415555 (10demon) >>! In T133968#2415541, @lfschenone wrote: > Thanks a lot demon! About lfs to lfschenone, do you mean we should wait... [19:03:19] coet|cawiki: heh, you are a bit instable today, hm? [19:34:40] zhuyifei1999_: sure [19:39:56] 10Quarry, 13Patch-For-Review: Add database selector - https://phabricator.wikimedia.org/T76466#2415766 (10Krenair) a:05Krenair>03None [19:41:51] !log phabricator deleting phab-01 instance due to it not starting. [19:41:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master [19:44:17] !log phabricator recreating phab-01 as a large size. [19:44:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master [19:52:41] !log phabricator installing apache2 and php5 on phab-02 [19:52:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master [19:53:15] 06Labs, 10Tool-Labs, 06Wikisource: "Bus error (core dumped)" error with "xvfb-run ebook-convert" - https://phabricator.wikimedia.org/T138976#2415811 (10Tpt) [20:06:32] zhuyifei1999_: I ran out of time, can you file a bug and CC me? I'll follow up on friday [20:06:47] k [20:10:08] actually I'll do later, gtg soon [20:27:25] 06Labs, 10Tool-Labs, 06Wikisource: "Bus error (core dumped)" error with "xvfb-run ebook-convert" - https://phabricator.wikimedia.org/T138976#2415811 (10Phe) on tools-exec-1204 : $ xvfb-run ebook-convert in.epub out.pdf works but on toolsexec-1404: $ xvfb-run ebook-convert in.epub out.pdf xvfb-run: error: Xv... [20:28:07] 06Labs, 10Phabricator: https://phab-01.wmflabs.org returns a core exception - https://phabricator.wikimedia.org/T137270#2363140 (10Dzahn) 13:24 < paladox> mutante https://phab-01.wmflabs.org/ 13:24 < paladox> I fixed phab-01 and recreated it on a large image [20:29:58] 10Labs-Other-Projects, 10Wikimedia-Bugzilla: bugs.wmflabs: bad gateway bugzilla.wmflabs: can't connect to db - https://phabricator.wikimedia.org/T138883#2416002 (10Dzahn) @Aklapper can you tell if we should fix or remove? [20:31:42] 06Labs, 10Phabricator: https://phab-01.wmflabs.org returns a core exception - https://phabricator.wikimedia.org/T137270#2416010 (10Paladox) a:05mmodell>03Paladox Reassining to me. Since I'm currently fixing the instance. [20:40:42] 06Labs, 06WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#1253259 (10chasemp) @d33tah are you still around and using this project? [20:43:40] 06Labs, 06WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#1253240 (10chasemp) @d33tah there is a banner that says `Note: this website is in alpha stage so far and might stop working anytime.` still on this site. Nothing this is from almost a year ago, is this s... [20:49:58] 06Labs, 06WMF-Legal: Request to review privacy policy and rules - https://phabricator.wikimedia.org/T97844#2416087 (10d33tah) W dniu 29.06.2016 o 22:43, chasemp pisze: > chasemp added a comment. > > > @d33tah there is a banner that > says Note: this website is i... [20:50:19] 06Labs, 06WMF-Legal: Discussion: can I park WikiSpy under a separate, simpler domain? - https://phabricator.wikimedia.org/T97846#2416088 (10d33tah) W dniu 29.06.2016 o 22:40, chasemp pisze: > chasemp added a comment. > > > @d33tah are you still around and > usin... [20:57:53] !log phabricator configured https correctly for phabricator at phab-01 [20:57:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master [21:05:27] 06Labs, 10Tool-Labs, 06Wikisource: "Bus error (core dumped)" error with "xvfb-run ebook-convert" - https://phabricator.wikimedia.org/T138976#2416140 (10Tpt) 05Open>03Resolved a:03Tpt The root cause of this issue was that the memory limit for job is now 256m, memory that is far lower than the one needed... [21:11:42] !log phabricator set mysql password @phab-01 [21:11:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master [21:14:24] ebernhardson: still need help w/ phab 2factor? [21:14:50] 06Labs, 10Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2416217 (10Paladox) [21:14:54] 06Labs, 10Phabricator: https://phab-01.wmflabs.org returns a core exception - https://phabricator.wikimedia.org/T137270#2416215 (10Paladox) 05Open>03Resolved Resolving this now. I had help with setting some things by @Luke081515 Please reopen if the problems continue. You will notice a performance incr... [21:16:01] !log phabricator phab-01 all setup now closing T137270 as resolved, uptime has improved now too :). [21:16:02]