[01:18:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:09:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:23:21] 06Labs, 10Tool-Labs: Create sqldump script - https://phabricator.wikimedia.org/T151680#2979087 (10scfc) >>! In T151680#2826140, @yuvipanda wrote: > Could we just rename replica.my.cnf to .my.cnf instead and provide > symlinks? :) +1 (not limited to this issue). `replica.my.cnf` is one of those solutions (IIR... [06:48:19] PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:28:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [09:29:40] 10Quarry: Explain command forces Quarry to keep running endlessly - https://phabricator.wikimedia.org/T155808#2955224 (10Nemo_bis) I'm still able to run new queries, so at least this doesn't seem to be disruptive for other users. Or do we have reasons to believe it is? [09:50:20] 10Quarry: Explain command forces Quarry to keep running endlessly - https://phabricator.wikimedia.org/T155808#2955224 (10zhuyifei1999) AFAIK, queries are automatically killed after some time. [09:57:06] 10Quarry: Explain command forces Quarry to keep running endlessly - https://phabricator.wikimedia.org/T155808#2979170 (10Soni) About a dozen or more queries that happened just after this query are still marked as "Running". I believe there are others as well, some more queries that followed these. https://quar... [10:30:09] Hi all [10:30:22] I have a problem with a server [10:31:22] After the 18th of January I have the /var/lib/puppet/clientbucket filled with 3.2 GB of files [10:31:49] Googling around I've found that can be a issue with NFS [10:32:29] Can I safely remove with a rm -rf /var/lib/puppet/clientbucket/* [10:33:03] ? [10:33:04] My server is osmit-tre [10:33:14] Thanks in advance [10:47:10] PROBLEM - Free space - all mounts on tools-exec-1221 is CRITICAL: CRITICAL: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-1221.diskspace.root.byte_percentfree (<44.44%) [12:22:16] 10Tool-Labs-tools-stewardbots: Outdated MySQL handling for hat-web-tool@stewardbots - https://phabricator.wikimedia.org/T156545#2979205 (10MarcoAurelio) [12:22:25] Steinsplitter: ^ [12:31:27] https://wikitech.wikimedia.org/wiki/Help_talk:Horizon_FAQ [12:37:05] 10Tool-Labs-tools-stewardbots, 07Need-volunteer, 07WorkType-Maintenance: Outdated MySQL handling for hat-web-tool@stewardbots - https://phabricator.wikimedia.org/T156545#2979228 (10MarcoAurelio) [12:47:29] Still no luck with https://www.mediawiki.org/wiki/MediaWiki-Vagrant either [12:53:48] 10Tool-Labs-tools-stewardbots, 07Need-volunteer, 07WorkType-Maintenance: SULWatcher: avoid logging account creations more than once - https://phabricator.wikimedia.org/T156546#2979231 (10MarcoAurelio) [12:54:24] 10Tool-Labs-tools-stewardbots, 07Need-volunteer, 07WorkType-Maintenance: SULWatcher: avoid logging account creations more than once - https://phabricator.wikimedia.org/T156546#2979244 (10MarcoAurelio) [12:57:14] 10Tool-Labs-tools-stewardbots, 07WorkType-Maintenance: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#2979247 (10MarcoAurelio) [12:57:58] 10Tool-Labs-tools-stewardbots, 10Pywikibot-General, 07Need-volunteer, 07WorkType-Maintenance: Evaluate cleanup on StewardBot's code - https://phabricator.wikimedia.org/T149404#2979248 (10MarcoAurelio) [12:58:15] 10Tool-Labs-tools-stewardbots, 10Pywikibot-General, 13Patch-For-Review, 07WorkType-NewFunctionality: StewardBot not logged into IRC - https://phabricator.wikimedia.org/T149265#2979250 (10MarcoAurelio) [13:03:57] 10Tool-Labs-tools-stewardbots (SULWatcher), 07Need-volunteer, 07WorkType-Maintenance: SULWatcher: avoid logging account creations more than once - https://phabricator.wikimedia.org/T156546#2979267 (10MarcoAurelio) [13:04:06] 10Tool-Labs-tools-stewardbots (SULWatcher), 07WorkType-Maintenance: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#2979269 (10MarcoAurelio) [13:04:10] 10Tool-Labs-tools-stewardbots (StewardBot), 10Pywikibot-General, 07Need-volunteer, 07WorkType-Maintenance: Evaluate cleanup on StewardBot's code - https://phabricator.wikimedia.org/T149404#2979271 (10MarcoAurelio) [13:04:12] 10Tool-Labs-tools-stewardbots (StewardBot), 10Pywikibot-General, 13Patch-For-Review, 07WorkType-NewFunctionality: StewardBot not logged into IRC - https://phabricator.wikimedia.org/T149265#2979273 (10MarcoAurelio) [13:07:26] 10Tool-Labs-tools-stewardbots (SULWatcher): IRC SUL Unification Bot Down - https://phabricator.wikimedia.org/T144887#2979278 (10MarcoAurelio) [13:09:35] 10Tool-Labs-tools-stewardbots, 10Pywikibot-General, 07Need-volunteer, 07WorkType-Maintenance: Evaluate cleanup on StewardBot's code - https://phabricator.wikimedia.org/T149404#2979286 (10MarcoAurelio) [13:09:40] 10Tool-Labs-tools-stewardbots, 10Pywikibot-General, 13Patch-For-Review, 07WorkType-NewFunctionality: StewardBot not logged into IRC - https://phabricator.wikimedia.org/T149265#2979288 (10MarcoAurelio) [13:09:46] 10Tool-Labs-tools-stewardbots, 07Need-volunteer, 07WorkType-Maintenance: SULWatcher: avoid logging account creations more than once - https://phabricator.wikimedia.org/T156546#2979292 (10MarcoAurelio) [13:09:48] 10Tool-Labs-tools-stewardbots, 07WorkType-Maintenance: Delete old data and/or stop logging to stewardbots' SULWatcher SQL DB - https://phabricator.wikimedia.org/T151113#2979294 (10MarcoAurelio) [13:09:50] 10Tool-Labs-tools-stewardbots: IRC SUL Unification Bot Down - https://phabricator.wikimedia.org/T144887#2979296 (10MarcoAurelio) [13:32:08] Uh, why does my instance ask for a password when I login [13:33:00] incorrect replica.my.cnf? [13:33:04] or not reading it? [13:33:15] TabbyCat: are you talking to me? [13:33:41] well, wikibugs is quite talkative but never answers to me, so yep :P [13:34:09] I'm however, out [13:34:16] lunch time [13:34:16] I'm talking about ssh to a non-tools machine [13:38:32] Maybe puppet hasn't run yet https://wikitech.wikimedia.org/wiki/Help:Access#Into_your-instance [14:02:50] Nemo_bis: asking password is really weird, iirc. password logins are disabled so if you don't have an ssh key it's simple permission denied. [14:12:02] https://github.com/wikimedia/operations-puppet/blob/0bf3f8a24fc3cf57ba86523d99869b4518d33f5f/modules/ssh/templates/sshd_config.erb#L83 [14:12:38] yeah probably puppet [14:16:18] This is what happens https://phabricator.wikimedia.org/P4827 [14:16:21] I'll wait a few hours [14:17:54] keyboard-interactive? is that 2fa? [14:25:35] Nemo_bis: https://phabricator.wikimedia.org/P4828 [14:26:15] keyboard-interactive is not supposed to be there at all. [14:27:02] https://github.com/wikimedia/operations-puppet/blob/0bf3f8a24fc3cf57ba86523d99869b4518d33f5f/modules/ssh/templates/sshd_config.erb#L136 <= it's only on iron for those with yubiauth (?) [14:31:53] does horizon have the instance console? wikitech used to have that, and on the fist boot you could check puppet progress from that console (it had only output though, no way to input anything) [14:36:12] No idea, I have no access to Horizon [14:38:32] o.O [14:38:48] you create instances from horizon right? [15:00:02] zhuyifei1999_: I don't [15:01:22] uh Special:NovaInstance don't work anymore iirc; where do you create instances besides from Horizon? [15:01:59] zhuyifei1999_: I used Special:NovaInstance [15:03:38] hmm, I just checked and apparently Special:NovaInstance still works o.O [15:03:59] you see a "get console output" link in the special page? [15:07:07] Yes, no info about puppet there yet [15:08:15] hmm, that's abnormal [15:14:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:12] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Xtools, 10DBA: `s51187__xtools_tmp` database using 272G on labsdb1001 - https://phabricator.wikimedia.org/T133321#2979483 (10jcrespo) p:05High>03Unbreak! This is creating large operational problems on labs due to lack of space. I had to delete all 2016* tables- i... [15:55:29] Is there something like quarry but for beta cluster? [15:58:02] that doesn't make much sense [15:58:17] if you want a nice GUI, there are things such as [15:58:32] https://dev.mysql.com/downloads/workbench/ [15:59:14] jynus, I want to access the DB some way. [15:59:59] mysql client is the easiest way to go [16:01:00] I'm not an idiot. At tools I can use sql enwiki for example. How can I do the same thing for beta enwiki? [16:01:14] I am not saying you are [16:01:23] I do not think that works on beta [16:01:55] I am saying you should use the mysql client directly [16:02:39] I guess running a mysql client at localhost won't help me. So where and with what params should I run it to access the db? [16:02:57] please do not guess [16:03:28] go to localhost and run it there, that's what I would do [16:03:51] This will give me access to the beta cluster db? [16:04:14] or run it from the app servers with the user configured there for the app (I do not know what it is) [16:06:34] Okay. If I want to query db for https://en.wikipedia.beta.wmflabs.org is there a public way how to do it? [16:06:52] sql enwiki_beta ? [16:06:56] idk [16:07:13] Could not find requested database [16:07:14] Make sure to ask for a db in format of _p [16:07:35] public, as in "everybody access" [16:07:46] I do not think so [16:08:00] I was assuming you had beta access [16:09:02] These are the users with access: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep [16:09:41] I have access to Toollabs only. This is the reason why we didn't understand each other. [16:09:49] ok, no [16:09:59] tools and beta have nothing to do with each other [16:10:00] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:10:21] and having access to tools give no access to beta [16:10:34] is there a reason you need it? [16:10:44] because it contains really no data [16:11:01] just "garbage" used for testing [16:11:35] Yes, there is a reason. I'd like to help to create a list of pages that should be undeleted. [16:12:17] not sure I follow you [16:12:44] beta is for mediawiki development [16:12:51] Yup, I know. [16:14:01] my confusion is that you seem to be talking about content [16:14:03] not code [16:14:28] I was asked to help with mass undeleting some pages deleted by a user by a Beta Wikimedia Steward. [16:14:42] ah, ok [16:14:53] then you need to apply for access there [16:15:27] And I need to create a list of pages that meets criteria I was told. This is the reason why I want SQL console. [16:16:13] just request to be a member of the project [16:16:32] beta normally is not problematic if there is a good reason [16:16:43] and that will give you access to the db [16:17:17] I guess this is a one time work rather than continuous work. Should I apply or ask a member? [16:17:45] (sorry for guessing again BTW) [16:18:00] anything, as long as it is properly logged (aka phab) [16:19:02] Not sure if you caught me. Should I apply for the membership or ask a current member for help? [16:19:12] *any of the 2 [16:19:49] just file a ticket for it [16:20:14] For what? [16:20:27] for either requesting membership [16:20:37] or asking for help to a current member [16:20:56] there is no labs-filtered-equivalent of beta [16:21:01] Ok, thanks. [16:22:16] BTW what is http://pastebin.com/CPh2xVbE then? [16:22:23] Why there is betawikiversity_p. [16:22:35] let me see [16:22:39] Ok [16:22:40] that is a good question [16:25:09] that is an actual production wiki [16:25:28] Why is a prod wiki prefixed with beta? [16:25:37] another good question :-) [16:25:50] at first I thought it was a mistake [16:25:54] but it is a real wiki [16:26:06] non-closed [16:26:10] public [16:26:31] maybe it was created before beta existed [16:26:54] https://beta.wikiversity.org/wiki/Main_Page ? [16:26:59] yep [16:27:41] you can see it is in the production config at: https://github.com/wikimedia/operations-mediawiki-config/blob/master/dblists/all.dblist [16:28:18] so maybe that strange happening confused you? [16:28:55] Yes, I saw something starting with "beta" so I asked how to get access to the other dbs. [16:29:44] ok, so who asked you to do that? because he/she may be confused, too [16:30:12] JustBerry [16:31:07] I am just saying that wikis like en.wikipedia.beta.wmflabs.org have nothing to do with the main cluster [16:31:13] no same credentials [16:31:15] etc. [16:31:30] Yes, I know. [16:31:37] maybe the petition is legitimate [16:31:47] but it will be harder to do, etc. [16:32:42] I wondering if it would make sense to replicate those to labs [16:33:16] As beta contain only random content it shouldn't make problems. [16:33:36] while I agree [16:34:04] I think in the past some people has expresed concen because it is also the place where security patches are tested [16:34:16] but probably not related to the db, only code [16:34:53] the other issue is that many times breaks, as it is its role [16:36:19] Can't be the code read in gerrit? [16:36:45] commited code can [16:37:02] remember beta is about testing it before merging it [16:37:33] again, I am not a beta expert, not my field [16:37:42] so maybe I am speculating [16:37:57] I would definitely file a ticket and see the easiest way to follow up [16:38:15] either providing you access or doing your request [16:40:30] I solved the problem by parsing the log already. [16:41:30] sorry I distract you by overthinking [16:43:03] No, you didn't distract me in any way. I only talk with you and help with JustBerry's problem (as I have no right at beta actually) at one time :) [16:46:58] 10Tool-Labs-tools-LTA-Knowledgebase: Localization - https://phabricator.wikimedia.org/T155738#2979540 (10Samtar) Working on moving most strings in PHP files to variables to assist with this in the near future [17:45:35] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Xtools, 10DBA: `s51187__xtools_tmp` database using 272G on labsdb1001 - https://phabricator.wikimedia.org/T133321#2979566 (10Matthewrbowker) a:05MusikAnimal>03Matthewrbowker The modifications made to the database code appear to not be working then... Taking this... [17:52:58] So... Why don't I have access to the Information_Schema for that database ^ [17:54:18] did you login with sql local? [17:54:44] Via Quarry. [17:55:08] oh, I think it's a private db [17:55:13] ah. [17:55:32] I'll just PHP it. [17:56:04] you could still create a cronjob to delete old rows, and I think that's the best way to do it right, since we can't seem to figure out why the code itself isn't cleaning up after itself [17:56:10] *right now [17:56:19] That's what I'm doing. [17:57:30] I'm sick of fixing this consistently broken code. So as we say at work: "Big hammer" [17:57:46] haha yeah me too [17:58:11] hopefully we'll be helping with the rewrite very soon [17:59:21] I hope so. [17:59:35] This is part of the reason I opted out of doing temp tables in the rewrite. [18:05:05] yeah I don't think it's necessary [18:21:26] 10Tool-Labs-tools-LTA-Knowledgebase, 10Wikimedia-Mailing-lists: Create mailing list for LTA Knowledgebase tool admins - https://phabricator.wikimedia.org/T156556#2979594 (10Samtar) [18:59:05] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Xtools, 10DBA: `s51187__xtools_tmp` database using 272G on labsdb1001 - https://phabricator.wikimedia.org/T133321#2979643 (10Matthewrbowker) p:05Unbreak!>03Normal Cleanup engine implemented. It's now running nightly at midnight via jsub and cron, Moving task to n... [18:59:33] 10Tool-Labs-tools-LTA-Knowledgebase, 10Wikimedia-Mailing-lists: Create mailing list for LTA Knowledgebase tool admins - https://phabricator.wikimedia.org/T156556#2979582 (10tomasz) Could you please provide the e-mail address for a secondary list administrator (as a backup) as per our [[https://meta.wikimedia.o... [19:18:35] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Xtools, 10DBA: `s51187__xtools_tmp` database using 272G on labsdb1001 - https://phabricator.wikimedia.org/T133321#2979667 (10jcrespo) I deeply thank the quick response. This was indeed a crisis. I will monitor the evolution, too. [19:41:24] hi, I noticed templatelinks table on labs have some entries that aren't shown in the wiki itself (there are no such transclusions) [19:42:03] looking on the page history, the page have never contained template, but there is an entry in templatelinks as it does contain [19:50:56] 06Labs, 10DBA: lab's templatelinks contain possible garbage rows - https://phabricator.wikimedia.org/T156560#2979685 (10eranroz) [20:00:28] 06Labs, 10DBA: lab's templatelinks contain possible garbage rows - https://phabricator.wikimedia.org/T156560#2979701 (10jcrespo) 05Open>03Resolved a:03jcrespo The main question here is if this is a labs issue or a production issue shown on labs. If it is production, that is more or less "normal", on a ne... [20:28:45] 10Tool-Labs-tools-Pageviews: Add page size column to Langviews - https://phabricator.wikimedia.org/T156562#2979717 (10MusikAnimal) [20:57:20] 10Tool-Labs-tools-LTA-Knowledgebase, 10Wikimedia-Mailing-lists: Create mailing list for LTA Knowledgebase tool admins - https://phabricator.wikimedia.org/T156556#2979748 (10Samtar) @tomasz sure - DatGuySteam@gmail.com [20:58:12] 10Tool-Labs-tools-LTA-Knowledgebase, 10Wikimedia-Mailing-lists: Create mailing list for LTA Knowledgebase tool admins - https://phabricator.wikimedia.org/T156556#2979582 (10DatGuy) Confirmed - Is me. [21:51:15] 10Tool-Labs-tools-LTA-Knowledgebase: Account management module [UI] - https://phabricator.wikimedia.org/T155348#2979804 (10DatGuy) Is this done now samtar? [22:01:36] Someone just tried to reset my wikitech account's password from ip 113.210.76.22. Just letting everyone know... [22:22:05] 10Tool-Labs-tools-LTA-Knowledgebase: Add WYMeditor - https://phabricator.wikimedia.org/T156566#2979830 (10Samtar) [22:29:52] 10Tool-Labs-tools-Pageviews: Create "Userviews" tool (probably can come up with a better name) - https://phabricator.wikimedia.org/T150585#2979845 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Deployed at https://tools.wmflabs.org/userviews [23:44:02] 10Tool-Labs-tools-Pageviews: Create tool description page on wikitech - https://phabricator.wikimedia.org/T142405#2979881 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Done! This was mostly just a description of how to setup and maintain the tools on Tool Labs, nothing more.