[12:23:53] volans: is there any way to tell cumin not to use color? [12:24:13] because this ain't so great: [12:24:17] $ nodeset -e $(sudo -H cumin 'A:db-section-s6 and A:eqiad' 2>&1 | head -n2 | tail -n1) [12:24:17] nodeset: Parse error: bad range: "cannot convert string to integer : "33mdb[1085"" [12:25:18] ah, haha. i see it's in the TODO file [12:31:48] `nodeset -e -S'\n' $(sudo -H cumin "$@" 2>&1 | head -n2 | tail -n1 | sed 's/\x1b\[[0-9;]*m//g') ` - this works [12:34:09] kormat: yes, you might be better of with 3 lines of python importing cumin and doing the query yourself [12:37:04] oh right, i forgot about that :) [12:40:15] volans: in fact i _already wrote_ that python script. sigh :) [12:40:51] yep we already talked about this [12:40:53] :) [12:45:07] phab requires 2fa to delete a comment i wrote? wow. [12:54:55] Well, it's trying to protect the world from you quietly erasing your dum-dums [12:56:46] hehe [12:59:00] based on results from https://gerrit.wikimedia.org/r/c/operations/puppet/+/626339 is https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Gerrit_integration out of date? probably jbond42 will know? [12:59:26] or maybe T166066 is what this is about? [12:59:27] T166066: Integrate the puppet compiler in the puppet CI pipeline - https://phabricator.wikimedia.org/T166066 [12:59:34] I am confused [12:59:42] jynus: i'm not sure which issues you're running into, [12:59:50] but `Hosts:` cannot be in the same block as Bug/Change-Id [12:59:57] it says I should not put it there [13:00:09] but validator does not like the extra space [13:00:16] commit msg validator on CI [13:00:59] hurm [13:01:12] jynus: the other thing is i'm not sure if you can put a space between hosts [13:01:13] maybe the issue got "fixed"? [13:01:32] i'd try `Hosts: db2141.eqiad.wmnet,db2097.codfw.wmnet` and see if that works [13:02:26] that is a valid suggestion, but what about the discrepancy between ci msg validator and documentation? [13:02:42] or you mean that is what was -1ing it before? [13:02:43] jynus: it could be the the validator is giving a bad error message [13:03:20] certainly doesn't hurt to try [13:04:21] jynus: the line the validator complained about was line 14, which is in the middle of the commit description [13:04:58] 14 was supposed to be the space between hosts and bug [13:05:20] https://gerrit.wikimedia.org/r/c/operations/puppet/+/626339/3//COMMIT_MSG ? [13:05:45] I don't think it counts those gerrit-only headers [13:06:10] ohh. _helpful_ [13:06:18] note it validated without the space, and I didn't touch the text [13:06:25] almost sure it is the newline [13:06:46] kormat: how do you do it that worked for you? [13:06:53] on other patches? [13:07:38] jynus: e.g. https://gerrit.wikimedia.org/r/c/operations/puppet/+/623567 [13:07:51] https://phabricator.wikimedia.org/T166066#6441831 [13:08:25] i asked legok.tm to do a new release of commit_message _validator [13:08:27] I can read that but not sure what that implies? [13:08:29] jbond42: ohh! progress [13:08:35] which means that Hosts now needs to be part of the foorter [13:08:37] outdated docs? [13:08:45] ok, thanks, that is what I was missing [13:08:49] I will update docs then? [13:08:53] i hadn't realised it would cause errors if it wasn't [13:08:56] please [13:09:09] in fairness, docs that are 2-days old are practically from the future compared to the norm around here ;) [13:09:21] yeah, not complaining about the docs [13:09:28] (or, for that matter, anywhere else i've worked :) [13:09:36] just was asking confirmation to update them myself [13:11:52] I was about to ask about limitation about comma separated items, but I will just try [13:12:42] these comments were not complains about docs, but thank you and permission to help by updating them myself [13:12:49] Jynus shuld be no limits however there is a limit on the line length so if you hit that use multiple Hosts: lines (not sure how the commit_validator) handles that [13:12:58] I see [13:13:05] I will add that useful tip too [13:13:10] you can also use a regex with `Hosts: re:.*` [13:15:22] https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Gerrit_integration [13:15:26] thanks, jbond42 [13:15:32] and kormat [13:15:52] np thanks for updating :) [13:16:56] and now that I have updated facter on puppet compiler for the new host, I think I can finally run it :-D [13:17:08] :D lol [13:20:02] sadly, I think it didn't trigger it [13:20:29] oh, it did [13:21:15] facts were not properly updated, retrying [13:22:13] but the trigger worked as intended, even with the extra spacee [13:24:13] party :) and yes pcc trims white space [14:05:46] _joe_: hey, remember how it was taking 15mins to grep for a given time range in the mw JobExecutor log file? [14:06:13] <_joe_> yes [14:06:27] https://phabricator.wikimedia.org/P12560 [14:06:42] equivalent search taking under 10s [14:06:59] <_joe_> wow [14:07:07] <_joe_> ok, puppetize it [14:07:11] <_joe_> so we can volint it [14:07:17] :D [14:07:25] <_joe_> (that is not mine, but I won't say the author) [14:07:50] no need, already done cat /home/kormat/bsearch.py [14:07:51] :D [14:08:17] <_joe_> kormat: so you used fh.readlines() as i suggested? :D [14:08:28] _joe_: suure [14:08:42] <_joe_> we just need 256 GB of RAM [14:09:16] Or sufficient amounts of swap. [14:09:27] * volans doesn't see an f.close() :-P [14:25:19] closing files manually is so 1999 [14:57:35] Duuude, everything is, like, in memory, y'know. [15:10:05] volans: any idea why `os.sendfile` would throw `OSError: [Errno 22] Invalid argument` if you tell it to use stdout? [15:10:26] kormat: in a meeting sorry, I'll get back to you shortly [15:11:21] stdout as in or out? [15:11:26] out :P [15:12:38] ah. it's the underlying libc throwing the error. `-1 EINVAL (Invalid argument)`. very helpful. [15:18:08] volans: i have a simple workaround, so it probably doesn't matter. [15:18:19] k [15:28:18] EINVAL Descriptor is not valid or locked, or an mmap(2)-like operation is not available for in_fd, or count is negative. [15:28:31] (from sendfile(2), the underlying syscall's man page) [15:28:53] there's one more option: `out_fd has the O_APPEND flag set. This is not currently supported by sendfile().` [15:29:09] which is my best bet, but i can't see a way to confirm/deny [15:29:49] doing `os.open("/dev/stdout", os.O_WRONLY)` gives me an fd that os.sendfile is perfectly happy with v0v [15:30:10] where does your out_fd come from? Just generic Python sys.stdout? [15:30:26] i was trying to use `sys.stdout.fileno()`, yeah [15:30:42] so 2 :D [15:30:50] >>> sys.stdout [15:30:52] s/2/1/ [15:30:52] <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'> [15:31:10] That indicates not-append (which would be (w+ or a, IIRC) [15:31:29] But this is a tty, what about your stdout? [15:31:37] klausman: a tty [15:31:43] so python isn't opening its own stdout [15:31:53] it just inherits it form the shell, with perhaps some fiddling with it [15:32:33] i can provide a minimal example, one sec. [15:33:04] you can change them ofc [15:33:09] * volans back [15:33:15] but I see you're already knee deep into it [15:33:21] probably I'm not needed :) [15:34:48] https://phabricator.wikimedia.org/P12562 [15:37:11] That works for me? [15:37:40] same for me (bash) [15:38:46] Also works in a pipeline [15:39:14] weeird. [15:39:22] what are you doing to your tty? :-P [15:39:25] i updated the paste with my kernel+python versions [15:39:43] I tried on cumin1001 [15:39:58] I've also tried on stat1005 [15:40:35] i get the same error (zsh+linux) [15:40:38] on macos I get [15:40:38] os.sendfile(out.fileno(), inf.fileno(), None, 16 * 1024) [15:40:38] TypeError: an integer is required (got type NoneType) [15:40:55] docs say None is specific to linux [15:40:57] with another file ofc, no fstab ;) [15:41:02] So for me that's 5.9.0-rc3 (sic!) and 4.19.0-10-amd64 for veriso, Python v3.7.3 and 3.7.9 [15:41:07] volans: change `None` to `0` [15:41:10] On Linux, if offset is given as None, the bytes are read from the current position of in and the position of in is updated. [15:41:22] kormat: OSError: [Errno 38] Socket operation on non-socket [15:41:25] paste updated [15:41:29] volans: hah [15:42:14] anyone know how to tell what open(2) flags are set on bash's stdout? [15:42:48] commented "something" [15:42:53] Not sure if flags is useful [15:43:43] flags is the same for stdin/out/err for me, so probably not [15:44:33] flags: 02 on prod fwiw [15:44:35] though the manpage does say it should show the open(2) flags [15:45:27] 02 on stat1005 as well, but it did not have a problem with the py script [15:45:59] I suspect that discrepancy between my WS and prod is more a matter of the kernel not providing info as opposed to an actual difference [15:47:02] `#define O_APPEND 00002000` [15:48:34] You could try and write a small C program that uses fcntl with F_GETFD to find the flags of your shell, to eliminate Python shenanigans [15:51:11] using python's fcnt.fcntl says the flags are 0 on all of stdin/out/err. i'm beginning to think they're special-cased somehow [15:52:48] Sorry, are you using F_GETFD? That is actually wrong, you want F_GETFL [15:53:40] in that case, the flags are all 1026 [15:54:41] https://phabricator.wikimedia.org/P12563 [15:54:51] Note that the proc output is in octal [15:55:46] ah, right. in that case what python says is the same as what /proc/self/fdinfo/1 says [15:56:04] which is O_APPEND|O_RDWR [15:59:18] anyway, nothing seems to be getting any clearer, so i'm just going to move on [16:02:48] I made one last update to my gist [16:02:56] paste* :) [17:16:01] ottomata: go ahead and merge my change when you get there [17:17:35] OH oops sorry andrewbogott [17:17:47] np I was just changing a code comment [17:36:05] so there is an issue on the api servers, we have an increase of 500 errors [17:36:23] https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors?_g=h@66534ad&_a=h@c92ef26 [17:36:34] points to Error 1205: Lock wait timeout exceeded; try restarting transaction [17:36:47] I am in a meeting so I am looking at things really slowly [17:37:56] https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?viewPanel=20&orgId=1&var-datasource=codfw%20prometheus%2Fops&var-cluster=api_appserver&var-method=GET&var-code=200 [17:39:46] it looks like the ReadingLists mw extension is hammering on the x1 db section [17:40:36] but that's about the extent of my insight [17:41:37] kormat: do you see info on a specific wiki ? [17:41:42] urm [17:41:47] sorry, it is enwiki [17:42:04] yeah - https://logstash.wikimedia.org/goto/61e27211c2a393503bc0032492b64693 says enwiki [18:04:00] helium, the former backup host (very old hardware) is being removed from ferm rules on all hosts right now. there will be ferm changes on puppet run on * [18:04:53] not forcing anything via cumin. just keeping an eye open for backup-related alerts