[00:48:08] 06Labs, 07LDAP, 13Patch-For-Review: Restore ldaplist -l passwd - https://phabricator.wikimedia.org/T122595#2392695 (10scfc) My WIP patch works in principle, and I'll add the Precise compatibility, //but//: It will fail for `ldaplist passwd` because there are `getent passwd | wc -l` = 3525 entries for users,... [03:52:02] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2392771 (10MZMcBride) The production `watchlist` table contains (wl_namespace, wl_title) entries that do not currently have `page` table (page_namespace, page_title) entri... [03:54:10] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2392772 (10MZMcBride) Example: has 1,952 page watchers, but this page does not... [03:55:51] anybody around from the Project-Admins group? [04:00:46] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2392773 (10MZMcBride) Hmmm, never mind. ``` MariaDB [enwiki_p]> select * from watchlist_count where wl_title = 'How_wikis_work' and wl_namespace in (0, 1); +----------+--... [04:03:29] 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: Make watchlist table available on labs - https://phabricator.wikimedia.org/T59617#2392774 (10MZMcBride) >>! In T59617#2392773, @MZMcBride wrote: > I guess my report changed for some other reason. There was a join with the `page` table due to page-move... [07:47:52] (03PS1) 10Ori.livneh: Add passwords::misc::private_static_site [labs/private] - 10https://gerrit.wikimedia.org/r/295193 [07:48:18] (03CR) 10Ori.livneh: [C: 032 V: 032] Add passwords::misc::private_static_site [labs/private] - 10https://gerrit.wikimedia.org/r/295193 (owner: 10Ori.livneh) [08:28:01] 06Labs: "No space left on device" on /data/scratch - https://phabricator.wikimedia.org/T138198#2392970 (10Smalyshev) [09:08:09] 06Labs, 07LDAP, 13Patch-For-Review: Restore ldaplist -l passwd - https://phabricator.wikimedia.org/T122595#2393032 (10MoritzMuehlenhoff) I've created https://gerrit.wikimedia.org/r/#/c/295198/ to raise the size limit. The reason my patch and our patch are not effective is because OpenLDAP handles paged searc... [10:28:55] 06Labs: "No space left on device" on /data/scratch - https://phabricator.wikimedia.org/T138198#2392970 (10zhuyifei1999) Ran out of inode: ``` 10:23:33 0 ✓ zhuyifei1999@tools-bastion-02: ~$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on udev... [13:15:53] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2393592 (10BBlack) >>! In T121279#2392449, @Merl wrote: > btw: All my code i licensed under MPL. Is the code available somewhere? I don't think... [14:47:54] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Intelligentsium was created, changed by Intelligentsium link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Intelligentsium edit summary: Created page with "{{Tools Access Request |Justification=Running my bot DYKReviewBot on the English Wikipedia. |Completed=false |User Name=Intelligentsium }}" [15:09:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Intelligentsium was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=666764 edit summary: [15:41:24] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries, 10DBA: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2393905 (10Soni) Just in case it helps, here's the output from a sample run of the Query - ``` SELECT page_id, page_title, first_rev_id -> FROM (SELEC... [15:53:43] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries, 10DBA: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2393960 (10jcrespo) Do not ever do `ORDER BY RAND()` on revision - you are literally reading all revisions, ordering all of them and then discarding the milli... [17:16:33] WTF is wrong with nVidia's DisplayPort support? It bluescreens when I use daisy chained and it use to work [17:23:29] 06Labs, 10Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2394274 (10mmodell) [17:23:33] 06Labs, 10Phabricator: https://phab-01.wmflabs.org returns a core exception - https://phabricator.wikimedia.org/T137270#2394272 (10mmodell) 05Open>03Resolved a:03mmodell [17:25:51] is anybody looking into /scratch on labstore being out of inodes? [17:26:15] !log tools.stashbot bounce stashbot [17:26:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [17:27:05] andrewbogott, yuvipanda: ^? [17:27:32] SMalyshev: yuvipanda and I are traveling, in theory chasemp might be able to look when he has a minute [17:27:49] andrewbogott: ok, thanks... [17:28:13] godog: is stashbot acting up? [17:28:57] bd808: I didn't see it online in irc, https://tools.wmflabs.org/sal doesn't have anything from today [17:29:04] hmm [17:29:31] SMalyshev: where do you see that reported? [17:29:49] chasemp: https://phabricator.wikimedia.org/T138198 [17:29:53] bd808: sorry if that was too trigger happy instead of actually see where it was stuck (if so) [17:30:24] godog: nope. I'll see if I can find any reason in the logs though [17:30:41] looks like it may have just netsplit or something? [17:31:28] godog: I'll make a note to backfill the missing logs too [17:33:40] bd808: thanks! does it deduplicate if it gets more backfilled logs than it needs? [17:34:46] godog: nope. it's dumb as rocks. I have some script on my home laptop that I've used to cut-n-paste data from the wikitech SAL into the right format for the elasticsearch backend [17:35:08] this would be a good time to document doing that [17:36:03] heheh I've been doing similar, but with my irssi logs -> wikitech SAL [17:36:41] yuvipanda: https://tools-prometheus.wmflabs.org/tools/targets is live again btw :D [17:36:54] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries, 10DBA: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2394317 (10Soni) Still getting an empty set. Not sure if the problem is with Query termination or the way the Query was framed :/ ``` MariaDB [enwiki_p]> USE... [17:43:44] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries, 10DBA: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2394336 (10jcrespo) `Empty set (16 min 47.81 sec)` on mysql command line client means that- no rows with the given filters. Your query is strange. This is th... [17:48:25] yuvipanda: grafana is broken again. ;-( [17:58:24] CP678|Laptop: I believe it is known and the host is down, somewhere tickets are in flight to remedy [17:59:45] chasemp: ok [19:53:19] 06Labs: "No space left on device" on /data/scratch - https://phabricator.wikimedia.org/T138198#2394904 (10chasemp) 05Open>03Resolved a:03chasemp All data in /scratch is ephemeral, though in practice we seldom have to clean house. I was forced to in this instance however and we are still in the process of... [19:58:28] 06Labs: "No space left on device" on /data/scratch - https://phabricator.wikimedia.org/T138198#2394928 (10Smalyshev) Hmm... Looks like by cleaning it up my home dir which was stored on scratch was erased too. That's not that huge of a deal by itself (even though unexpected) but now I can't even log in to my inst... [19:58:47] chasemp: could you look into https://phabricator.wikimedia.org/T138198#2394928 ? [19:59:22] 06Labs: "No space left on device" on /data/scratch - https://phabricator.wikimedia.org/T138198#2394929 (10chasemp) why would your home dir be on scratch? [19:59:39] SMalyshev: I don't understand the how of that [19:59:52] you are saying /home/smalyshev was a link to /data/scratch? [19:59:58] chasemp: yes [20:00:33] because it had the space I periodically need to keep stuff, and is shared between wikidata-query project instances [20:00:35] that's not a great idea I think for this reason, scratch is by nature things we delete w/o questinging [20:00:43] ok [20:00:52] well, that's the only shared space I've got since shared home got nuked [20:01:19] if you can suggest a better shared space I could switch to that definitely [20:01:41] a good thing to do w/ would be a per project /data/project share, that's typical [20:02:12] in general though, NFS as a file share in this way just isn't going to work out very well I think [20:02:13] I remember somebody (Yuvi?) suggested not to use that. Don't remember why exactly, maybe something to do with NFS settings [20:02:16] is this for like...deployments? [20:02:34] chasemp: for test deployments, in particular, yes. And for other stuff too [20:03:07] chasemp: do we have other way of making file share besides NFS? [20:03:49] we don't, but then the usual tag line is, we don't have NFS in prod and we do ok, nfs for deployments etc is pretty sketch in general [20:04:23] there are conveniences to it for sure but our setup is one where every problem has seemed like a nail for the nfs hammer and thus instability and some amount of pushback [20:04:24] chasemp: it's not production, it's a workspace... so I need some shared space that I can use between hosts [20:04:57] e.g. for shared configs, shared files, shared data, etc. [20:05:32] (wouldn't hurt for production too, so I don't need to re-process 10G dumps on each host individually, but it's rare so I don't have enough motivation about it) [20:05:36] I understand, I'm hoping to explain the general expectations for NFS [20:05:44] what we really need for things like that is a better way to have per-project Puppet config [20:06:12] local puppetmaster is certainly possible but it's harder to deal with than it should be [20:06:14] bd808: how puppet config would help? [20:06:34] SMalyshev: puppet is the better way to do shared configs [20:06:44] the shared things you mentioned all sounded like Puppet stuff [20:06:52] using nfs for configuration distribution, deployments, etc is bad really [20:06:55] is where I was going [20:06:56] bd808: they are not [20:07:14] bd808: you don't really want wikidata dump to be checked into puppet :) [20:07:33] heh. probably not, no [20:07:41] anyhoo, a reasonable thing is a /data/project allocation like tools uses so it's not locked up or tied to a single user home dir [20:07:54] no but that dump served form teh dumps server is probably practical [20:08:23] but in general I would shy away from linking your home to NFS scratch as it will get nuked periodically [20:08:29] and it would never be expected [20:08:30] chasemp: that's not that dump, it's processed. But anyway, let's not get too far into the woods, the gist of it that there's a need for project-wide shared space on labs [20:08:53] we have that with NFS. it is just disabled by default now [20:09:03] I think in most cases there isn't is what both of us are trying to convey, it's just convenient not necesssarily good [20:09:07] and yes we can sort it out if it makes sense [20:09:30] chasemp: so how would you propose to share data then? [20:09:52] what are you doing? I don't know much about it [20:10:13] this specific project is for wikidata query service [20:10:42] it sounds like some of the NFS shares should just be turned back on for that project [20:11:00] I think it is possible to do that with hiera config on wikitech [20:11:09] it's not actually but we can do it [20:11:33] it's locked up in a yaml file in puppet to make it ops only from previouse cleanup / abusive uses [20:11:35] well, I was using /data/scratch I think it was on Yuvi's advice but I may be mistaken [20:11:46] the downside of adding NFS back to a project is that if the nfs server has issues it may effect your project [20:12:00] but if you enable project space (disabled currently AFAIK) I can use that instead [20:12:17] provided it doesn't mind occasional 20G dump [20:12:50] bd808: the runtime doesn't use NFS, it's used to transfer/deploy/store files [20:12:59] so that [20:13:04] so that's not a problem [20:13:22] there are a few threads here, don't link your home to scratch that'll just break at some point, 2. scratch is cool to use but it's ephemeral not long lived by nature, 3. relying on nfs for deployments or general thigns we don't use it for in production is considered bad and please try not to do it, 4. if you have dumps esque share data for your project sure we can roll out a /data/project share for it [20:13:28] no worries, toss a ticket my way but may take a few days [20:13:36] we are very short handed, i.e. just me this entire week [20:13:36] but a more pressing problem is that I can't log in to any of these hosts anymore, so what do I do? [20:13:48] try wdq-beta.wikidata-query.eqiad.wmflabs [20:14:09] chasemp: ok, now works [20:14:22] what others instances are in this project that you changed this on? [20:14:32] i.e. if this was a puppetized setting we could just unroll w/ puppet [20:14:38] that's the kind of thing bd808 and I were getting at [20:14:42] but it's all good just a thought [20:15:23] chasemp: wdq-deploy, wdq-varnish and wdqs-puppetmaster probably [20:16:28] chasemp: I'm not sure how I could puppetize it given that there's single puppet reporitory (which is also used for production) and I may need to tweak stuff rather often [20:16:49] I don't want my experiments on labs be mixed with production setup [20:16:49] the use case now is to run a puppet master in your project that rebases off the main one [20:17:15] that's a fair thought ubt it's a trade off either way because here we are fixing all of them by hand [20:17:27] chasemp: I do. but I'm not sure how that solves my issues... [20:17:52] so wdq-deploy seems not there, puppetmaster either, but varnish seemed ok [20:18:01] especially as puppetmaster is broken in the same way now, so I'd be unable to fix it anyway [20:18:02] want to give the ones you know about a try across? [20:19:06] $ ssh wdqs-puppetmaster.eqiad.wmflabs [20:19:08] Creating directory '/home/smalyshev'. [20:19:08] Unable to create and initialize directory '/home/smalyshev'. [20:19:40] so puppetmaster is broken. wdq-varnish in fine [20:20:11] SMalyshev: this is like a 5 sided convo so some parts of it seem confusing, the idea in question is that you changed this manually for your home directory when in production or any system taken seriously that should survive more than a day it should be done in a repeatable manner we can change and rollback in this case pupppet [20:20:42] it's a question to me and maybe us as labs as to how you can best set yourself up for success, and that is puppetizing changes as you would otherwise [20:20:54] and how that lends itself to less NFS usage as you don't need nfs to distribute configurations [20:21:01] that's the gist of that side bar [20:21:19] chasemp: but it's not a production system. It's labs system for testing all kinds of stuff. And I'm not sure how I could puppetize that... [20:21:32] well here you are asking for support on it [20:21:42] wdq-deploy.eqiad.wmflabs seems to be just stuck - not answering for ssh at all [20:22:01] I can't get on that either so it may be toast for other reasons? [20:22:33] I'll try to reboot it [20:23:30] I get that in your mind it seems like a toy system for testing things but you are also expecting some level of support and tiem from us it seems, and that's ok [20:23:40] but it's either a toy or something we want to be repeatabile and support [20:23:44] it cannot be both [20:23:54] so respectfully just try to do things using puppet so it's easier on us [20:23:59] that's all I'm trying to say [20:24:00] chasemp: well, I don't really expect much above "I put stuff somewhere, it stays there" [20:24:24] well, that and "I can share files between two hosts" :) [20:25:18] I don't understand the use case still I think, if it was for deployments I would say let's not if we can avoid it, if it's some kind of dumpm that needs to be available project wide I totally get it and ok [20:25:24] drop a ticket on me and I'll look at adding it [20:25:27] it's been awhile [20:28:07] 06Labs: Add project-wide shared space for wikidata-query project - https://phabricator.wikimedia.org/T138259#2395016 (10Smalyshev) [20:28:25] chasemp: ok, wdq-deploy seems to be fine, Filed T138259 for enabling shared space for wikidata-query [20:28:26] T138259: Add project-wide shared space for wikidata-query project - https://phabricator.wikimedia.org/T138259 [21:11:43] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2395234 (10scfc) [21:11:45] 06Labs, 10Tool-Labs, 13Patch-For-Review: Some Tools users do not show up in create-dbusers query - https://phabricator.wikimedia.org/T138102#2395233 (10scfc) 05Open>03Resolved [21:50:22] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2395294 (10scfc) [21:50:24] 06Labs, 10Tool-Labs: No 'replica.my.cnf' in my home directory at maos@tools-bastion-03 - https://phabricator.wikimedia.org/T137283#2395292 (10scfc) 05Open>03Resolved a:03scfc [21:51:07] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2315661 (10scfc) [21:51:09] 06Labs, 10Tool-Labs: /home/ium missing replica.my.cnf - https://phabricator.wikimedia.org/T134066#2395295 (10scfc) 05Open>03Resolved a:03scfc [21:51:46] 06Labs, 10Tool-Labs: No 'replica.my.cnf' in my home directory at maos@tools-bastion-03 - https://phabricator.wikimedia.org/T137283#2363528 (10scfc) You now have a working `replica.my.cnf` in your home directory. [21:52:14] 06Labs, 10Tool-Labs: /home/ium missing replica.my.cnf - https://phabricator.wikimedia.org/T134066#2253830 (10scfc) You now have a working `replica.my.cnf` in your home directory. [21:52:52] 06Labs, 10Tool-Labs: Missing replicy.my.conf on Tools Labs - https://phabricator.wikimedia.org/T138037#2395300 (10scfc) 05Open>03Resolved a:03scfc You now have a working `replica.my.cnf` in your home directory. [22:00:20] 06Labs, 10Tool-Labs: Reset replica.my.cnf for y-verciti - https://phabricator.wikimedia.org/T138270#2395306 (10scfc) [22:02:26] 06Labs, 10Tool-Labs: [Tool Labs] Database credential file replica.my.cnf missing in my home directory on Tool Labs (/home/wiki13). - https://phabricator.wikimedia.org/T122657#2395331 (10scfc) [22:03:50] 06Labs, 10Tool-Labs: /home/tanvir missing replica.my.cnf - https://phabricator.wikimedia.org/T131546#2395332 (10scfc) [23:57:09] 06Labs, 10Tool-Labs: Restore replica.my.cnf for toolsbeta.admin - https://phabricator.wikimedia.org/T109807#2395493 (10scfc)