[01:10:20] (PS1) Adamw: WIP PHPUnit tests under Drupal [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/120735 [01:11:14] #1454: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1454 [01:24:40] (PS3) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [01:25:01] (CR) Adamw: [C: -2] "I still need to test the recurring drush job" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 (owner: Adamw) [01:37:21] (PS4) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [01:40:49] (PS1) Adamw: Decouple recurring and one-time queue processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/120742 [01:41:24] (PS5) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [01:41:43] (PS6) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [01:45:12] (PS1) Adamw: Patch dependency glitch in the dequeue loop [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/120743 [01:46:54] (PS7) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [01:48:02] (PS1) Adamw: Fix a bug which would cause a failure to report how many messages were processed [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/120744 [01:49:43] (PS8) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [01:50:00] (CR) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 (owner: Adamw) [01:50:20] (PS2) Adamw: Decouple recurring and one-time queue processing [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/120742 [01:50:25] (PS2) Adamw: Fix a bug which would cause a failure to report how many messages were processed [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/120744 [01:50:30] (PS9) Adamw: (FR #1436) Add a job time limit to queue processing modules [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/119546 [01:54:14] #1463: (AW) TS:PD -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1463 [01:54:14] #1463: (AW) *Deployed* -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1463 [01:54:14] #1463: (AW) The logging patch was not deployed, but meh. -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1463 [01:54:14] #1487: (AW) TS:PCR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1487 [01:54:15] #1487: (AW) *Deployed* -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1487 [01:54:15] #1487: (AW) TS:PD -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1487 [01:56:14] #1493: (AW) O:AW|TS:B|P:MH|TT:CT|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1493 [01:57:14] #1493: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1493 [06:17:14] #1447: (AG) *Deployed* -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1447 [06:17:14] #1447: (AG) Oops :) TS:PD|MtDo:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1447 [11:06:09] what is Adam Wight email ? [16:07:17] (PS1) Adamw: Configuration to control stdout quietness [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/120810 [16:08:56] Jeff_Green: ^^ [16:09:07] chanhop. [16:09:15] it's nicer over here, anyway. [16:10:04] sometimes I feel like boycotting the privchan entirely. [16:12:58] funny, for me it's usually about boycotting gerrit [16:13:00] :-P [16:13:47] lol. I wish I could say I had an evil plan to revive Special:Code using Extension:Annotator. [16:13:59] oops, did I say that? [16:14:16] awight: it looks fine to me, but I'm gonna leave it to qualified professional coders to ++ it [16:14:22] shore [16:14:37] I gotta reboot, my laptop is freaking out after a giant mess of package updates [16:14:41] * awight blushes at almost being included in the "qualified" set [16:15:17] pizzzacat1: want some anguish-inducing CR? [16:15:23] pizzzacat1: https://gerrit.wikimedia.org/r/120810 [16:48:28] (CR) Ssmith: [C: 2 V: 2] Configuration to control stdout quietness [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/120810 (owner: Adamw) [16:51:56] Jeff_Green: when you do the crontab spelunking, please add the "quiet: 1" line to each relevant config file. [16:53:05] !log update tools from 2cfc441a9e5d2de3cc892d0077603579b220ac0e to 400b4f6b0cf69ac0a7f3621f568f1707f5b4aa85 [16:53:12] Logged the message, Master [17:06:15] #1481: (C) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1481 [17:54:50] awight|mailbox: Is that dedupe cronspam normal now? [17:58:57] (PS4) Mwalker: WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 [18:00:10] (CR) jenkins-bot: [V: -1] WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [18:08:18] K4-713: no it will go away once we push a config change [18:08:38] Jeff_Green: ^^ will u have a sec to do that? [18:08:50] * awight applies "electric ping" [18:14:23] K4-713: I'm probably going to be bouncing on and off here, got a sick kid and dog is wearing a bunch of stitches, drain, collar... thus i have duties. But, can you catch Jeff_Green some time today and ask for the "quiet: 1" config? [18:14:41] awight: I'll see what I can do, yes. [18:15:01] thx [18:21:15] #1445: (C) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1445 [19:03:54] awight: hello [19:04:05] matanya: good morning! [19:04:19] you too :) regarding stat1 access, do you need it ? [19:05:08] oh hmm I think that was a very long time ago... Remind me what is done there? [19:05:15] That is required for hadoop queries, right? [19:05:18] analytics [19:05:42] Yes I have things I would like to do there, to start with, I want to help with instrumenting page save conflicts... [19:05:43] you haven't logged in since 2012 [19:05:50] oh hah [19:05:53] I believe that [19:06:12] the new host for that will be stat1003 [19:06:26] I think the new anal cluster was still experimental at that point and perhaps I couldn't do what I needed yet. [19:06:43] matanya: well, let me know what I should do to help migrate the account [19:06:50] can you move the stuff you need from there, so i can remove your access [19:07:06] sure. done! [19:07:06] did you see the mail by otto ? [19:07:18] matanya: nope, looking now [19:07:24] great [19:07:28] thank you very much [19:07:30] "stat1 account audit [19:07:30] wm-lists x [19:07:33] k [19:07:58] thanks for the heads-up! [19:21:36] Jeff_Green: erm, I just found a weakness in the quiet:1 thing we were discussing. If a cronjob fails, the fail email will not have the verbose log messages. [19:22:05] ah yeah [19:22:07] Is it possible to set cron up to email only if stderr was written, but include all of stdout? [19:22:41] yes [19:23:08] awight: iirc it will mail if it receives either stdout or stderr [19:23:42] the easiest way i know of is redirct stdout to file and attach it [19:23:54] matanya: nice workaround! [19:24:00] ha the horror [19:24:16] ugly, but works [19:24:24] hehe, luckily I am already teeing error-level messages to both out and err, so that would work perfectly in our case [19:24:46] i can't wait to see the shell jankery in the new version of the wrapper [19:25:09] lol, what is a shell script if not jankery... [19:25:27] wait, why even print any output? why not just make the script generate the mail? [19:25:37] that's sort of horrible [19:25:43] this isn't just the drush script... [19:25:52] it's also tools/ stuff, and stray mysql scripts [19:26:05] why is it horrible? [19:26:08] mailx -s will work just fine if i understand the secanrio [19:26:11] cos we'd need multiple wrappers [19:26:34] how do you not already need them? are you wrapping the wrapper or something? [19:27:21] I think we want generic behavior, where we run any thing that emits stdout and stderr [19:27:39] urp. This sort of belongs in our jenkins-replacement doc [19:27:44] * awight rustles a bit [19:28:28] https://docs.google.com/a/wikimedia.org/document/d/1UkfeQFvOQ0FVLdNJN1rJSEVPwCOoXeIb8wJ2AyBY5hc/edit?usp=drive_web [19:28:48] We should probably make that publicly readable, eh? [19:29:14] this reads like a document to make it impossible to replace jenkins :-P [19:29:20] "ram/cpu limits" [19:29:33] jenkins doesn't have does it? [19:29:59] lol [19:30:21] yeah we should define the MVP [19:30:43] Job Management "free up 3GB of RAM" DONE! [19:31:07] Serious. [19:31:24] I was actually pleased by something jenkins, the other day. [19:31:36] so am I correct that you're writing a grand unified wrapper? [19:31:51] I watched resource usage during job run and it manages to avoid spawning another java thread when forking the job runner [19:31:56] Jeff_Green: erm no? [19:32:08] I could if that becomes a thing. [19:32:15] We still need some buy-in from K4|lunch IMO [19:32:51] and... maybe atgomez|solunch too [19:32:54] i'm adding snide comments to the doc [19:32:58] sweet ;) [19:33:26] What about using formatting or moving stuff to make the MVP jump out? [19:34:02] have u thought upon how we can share editing of a crontab? [19:34:07] well [19:34:14] as I see it we have conflicting demands [19:34:28] not like ops vs coders though [19:34:29] mwalker: u might be interested [19:34:47] whats up? [19:34:47] we want easy quick editing, and we want it to be tamper-proof [19:35:01] mwalker: backscroll: cron vs jenkins [19:35:06] https://docs.google.com/a/wikimedia.org/document/d/1UkfeQFvOQ0FVLdNJN1rJSEVPwCOoXeIb8wJ2AyBY5hc/edit?usp=drive_web [19:35:25] no one asked me, but i advise on jenkins [19:35:32] eh? [19:35:43] meaning, you think we should stick with jenkins? [19:35:46] yes [19:35:54] for now yeah but we're talking about something better [19:35:55] as long as any of this is for in-scope systems (payments, payments-listener for sure) easy-editing by lots of people is not necessarily desireable [19:36:00] without access to the doc though [19:37:03] matanya: cool, glad to hear you're interested! I've made the doc public [19:37:28] thanks [19:37:34] Jeff_Green: yeah cron++ will give us much more granularity [19:37:44] granular ACL i mean [19:38:34] awight: i'm not sure if you're serious? :-) [19:38:50] ah i see now [19:39:01] jenkins has ACL too, i guess you know that [19:39:19] matanya: oh! now I know ;) [19:39:29] but all the jenkins stuff runs as the same user right? [19:39:38] doesn't have to [19:39:45] if you run jenkins as root? [19:39:46] you can seperate the runners [19:39:51] Jeff_Green: what this doc is missing is a paragraph motivating why we want to do something new [19:39:55] matanya: oic [19:40:04] no, you can define a slave to run as user x [19:40:10] rad. [19:40:44] matanya: you know if anyone has succeeded in puppetizing Jenkins, beyond jjb-config tools? [19:41:08] awight: well, i did :) but not for wmf [19:41:14] #1203: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1203 [19:41:14] #1201: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1201 [19:41:14] #1202: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1202 [19:41:26] awight: maybe rather than "why do anything different" it should be "what are our needs and are they being addressed by the current system"? [19:41:29] the wmf jenkins stuff is done in three places, by hashar [19:41:30] matanya: you could suddenly come out with a very similar open-source version ;) [19:41:42] Jeff_Green: yes! something along those lines. [19:42:01] matanya: I do know about zuul and jjb-config [19:42:03] i wish i could awight :) [19:42:03] jenkins, for one thing, is not very visible [19:42:19] unless you want to regularly log in to jenkins [19:42:48] matanya: I think you are allowed to share the outline of something similar you thought of just now [19:43:04] it doesn't log usefully, no central logging for example, and also it is incompetent at serving up its own log data [19:43:20] Jeff_Green: that can be changed too [19:43:30] you can have a central logging server [19:43:45] matanya: syslog? or a jenkins-specific one? [19:43:47] and even query them and mail alrets etc [19:43:52] both [19:44:14] as it is we have to roll off the jenkins logs with an external script because it can't function with too much log data [19:44:14] #1210: (AG) BG:#SPME(i|T:BC -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1210 [19:44:14] #1195: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1195 [19:44:14] #1150: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1150 [19:44:14] #1197: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1197 [19:44:14] #1196: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1196 [19:44:15] #1151: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1151 [19:44:15] #1153: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1153 [19:44:16] #1155: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1155 [19:44:16] #1148: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1148 [19:44:17] #1149: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1149 [19:44:17] #1152: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1152 [19:44:18] #1021: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1021 [19:44:18] #1156: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1156 [19:44:19] #1157: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1157 [19:44:25] ha go slander go [19:45:15] #1245: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1245 [19:45:15] #1130: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1130 [19:45:15] #1253: (AG) BG:#IT|T:BC Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1253 [19:45:15] #418: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/418 [19:45:15] #698: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/698 [19:45:15] #676: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/676 [19:45:15] #783: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/783 [19:45:16] #933: (AG) BG:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/933 [19:45:32] at $day_job jenkins steams some of his logs to a central logging server, and that one is queried by logstash and graphite and all that jazz [19:47:31] so Jeff_Green all is doable, just java crap, and time consuming [19:49:56] matanya: yeah, it's sort of a cascading dependency mess to make it 'good' [19:50:34] took me about a month until i was pleased with it, but now it is like a charm [19:51:05] don't get me wrong, I understand that it has a role in the world. i just don't see it as being great for a cron replacement [19:51:27] yeah, sure [19:51:39] the doc will surely clarify this [19:52:26] matanya, so; you built a bunch of jenkins extensions to correct your deficiencies? or did you find a bunch of premade ones that fulfilled your needs? [19:52:42] the latter [19:53:01] there are like a million jenkins add-ons [19:53:15] matanya: can u estimate how long you spent tweaking Jenkins? [19:53:33] about two weeks for all 4 env we have [19:53:40] nice, thx [19:54:00] for main stuff, nice to-haves to a bit more [19:54:08] *took [19:55:13] the flow is : read source code, test in staging, push to prod. be happy [21:37:11] (PS1) Adamw: Delay logging during config startup, to avoid stdout when quiet=1 [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/120943 [21:38:30] pizzzacat: ^^ ! [21:39:48] The ship is... almost seaworthy? [21:39:48] * awight continues bailing furiously [21:41:06] (PS1) Katie Horn: Fixing a problem with ewallets, rtbt, and boletos [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/120945 [21:45:28] K4-713: want CR yet? [21:45:28] If you're offering, yes. This should be easy: I'm just resurrecting a function I killed somehow. [21:45:28] And adding tests to make sure it doesn't get killed again. [21:45:28] ...mostly the latter. [21:45:29] gerrit is a DOG today [21:45:29] Arf [21:45:29] hehe last thing you want to do is impersonate a gerrit. Gerrit would be lucky if we just sent the dogcatcher after it. [21:45:55] (CR) Adamw: "Just one question" (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/120945 (owner: Katie Horn) [21:46:29] (PS5) Mwalker: WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 [21:46:36] (CR) jenkins-bot: [V: -1] WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [21:46:38] (CR) Adamw: [C: 2 V: 2] Fixing a problem with ewallets, rtbt, and boletos [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/120945 (owner: Katie Horn) [21:47:01] Can anyone CR https://gerrit.wikimedia.org/r/120943? [21:47:21] awight: Argh: My formation! [21:47:31] :p [21:47:56] If only there was a unit test for typos that made other words in comments. [21:47:58] hehe [21:48:18] Sorry for the hasty merge, I just decided it doesn't matter. [21:48:28] (CR) Mwalker: [C: 2] Delay logging during config startup, to avoid stdout when quiet=1 [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/120943 (owner: Adamw) [21:48:32] thx [21:49:29] !log updated tools from 400b4f6b0cf69ac0a7f3621f568f1707f5b4aa85 to 0eb485c8b6db5f06805976860bce7aa8b0d6444b [21:49:35] Logged the message, Master [22:07:31] (PS2) Katie Horn: DO NOT MERGE: Adding a component so we can bypass the visual stuff for batch operations and api requests that don't need a form. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/119218 [22:11:24] (PS3) Katie Horn: Adding a component so we can bypass the visual stuff for batch operations and api requests that don't need a form. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/119218 [22:13:27] (PS6) Mwalker: WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 [22:13:33] (CR) jenkins-bot: [V: -1] WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [22:23:04] (CR) Mwalker: [C: 2] "I like it!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/119218 (owner: Katie Horn) [22:23:43] (PS7) Katie Horn: WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [22:24:04] (CR) jenkins-bot: [V: -1] WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [22:27:21] (PS8) Mwalker: WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 [22:27:43] (CR) jenkins-bot: [V: -1] WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [22:28:01] Okay, our cronspamhose should be turned off, now. [22:49:34] (PS9) Katie Horn: WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [22:49:41] (CR) jenkins-bot: [V: -1] WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [22:51:02] (CR) Katie Horn: "PS9 just alters the test credentials for WorldPay such that the unit tests can pass now." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker) [22:52:49] (PS10) Mwalker: WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 [22:52:54] (CR) jenkins-bot: [V: -1] WIP: Bigger stub for worldpay [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/118647 (owner: Mwalker)