[01:23:11] Fundraising-Backlog: process-control should not crash on bad utf-8 from stdout or stderr - https://phabricator.wikimedia.org/T167849#3346812 (Ejegg) [01:59:07] (CR) Reedy: [C: 2] API: Split description messages into summary + additional text [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/358677 (owner: Anomie) [02:01:47] (Merged) jenkins-bot: API: Split description messages into summary + additional text [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/358677 (owner: Anomie) [02:09:55] (CR) Reedy: [C: 2] Make phpcs pass - special / tests [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358044 (owner: Umherirrender) [02:17:26] (PS1) Ejegg: Handle invalid utf-8 coming from stderr/stdout streams [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/358900 (https://phabricator.wikimedia.org/T167849) [02:20:52] Fundraising Sprint Kickstopper, Fundraising-Backlog, Patch-For-Review: process-control should not crash on bad utf-8 from stdout or stderr - https://phabricator.wikimedia.org/T167849#3346850 (Ejegg) p:Triage>High a:Ejegg [02:30:33] Fundraising Sprint Kickstopper, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Instead of failmail, record and flag duplicate donations. - https://phabricator.wikimedia.org/T167520#3335638 (Ejegg) a:mepps [03:17:37] (CR) MtDu: "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [03:19:44] (CR) jerkins-bot: [V: -1] API: Split description messages into summary + additional text [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [07:24:17] (PS2) Umherirrender: Make phpcs pass - special / tests [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358044 [07:24:27] (CR) Umherirrender: [C: 2] "Resubmit" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358044 (owner: Umherirrender) [07:28:18] (Merged) jenkins-bot: Make phpcs pass - special / tests [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358044 (owner: Umherirrender) [07:28:50] (PS2) Umherirrender: Add phpcs and make pass [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358045 [07:31:08] (CR) Thiemo Mättig (WMDE): [C: 1] "Just one question/suggestion, +2 otherwise." (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358045 (owner: Umherirrender) [07:31:52] (CR) Thiemo Mättig (WMDE): [C: 2] Use short array syntax in alias file [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358048 (owner: Umherirrender) [07:39:29] (CR) Umherirrender: Add phpcs and make pass (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358045 (owner: Umherirrender) [07:40:06] (PS3) Umherirrender: Add phpcs and make pass [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358045 [07:46:38] (CR) Thiemo Mättig (WMDE): [C: 2] Use short array syntax - special [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358049 (owner: Umherirrender) [07:47:25] (CR) Thiemo Mättig (WMDE): [C: 2] Use short array syntax - includes [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358050 (owner: Umherirrender) [07:47:49] (CR) Thiemo Mättig (WMDE): [C: 2] Use short array syntax - api / tests [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358051 (owner: Umherirrender) [07:48:21] (CR) Thiemo Mättig (WMDE): [C: 2] Use short array syntax [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358052 (owner: Umherirrender) [07:51:27] (CR) Thiemo Mättig (WMDE): [C: 1] Make phpcs pass - includes (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358043 (owner: Umherirrender) [07:59:22] (CR) Umherirrender: Make phpcs pass - includes (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358043 (owner: Umherirrender) [08:12:08] (PS3) Umherirrender: Use short array syntax in alias file [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358048 [08:12:15] (CR) Umherirrender: [C: 2] "Resubmit" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358048 (owner: Umherirrender) [08:12:34] (PS4) Umherirrender: Use short array syntax - special [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358049 [08:12:43] (CR) Umherirrender: [C: 2] "Resubmit" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358049 (owner: Umherirrender) [08:13:29] (PS4) Umherirrender: Use short array syntax - api / tests [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358051 [08:13:37] (CR) Umherirrender: [C: 2] "Resubmit" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358051 (owner: Umherirrender) [08:19:57] (Merged) jenkins-bot: Use short array syntax in alias file [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358048 (owner: Umherirrender) [08:19:59] (Merged) jenkins-bot: Use short array syntax - special [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358049 (owner: Umherirrender) [08:20:01] (Merged) jenkins-bot: Use short array syntax - api / tests [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358051 (owner: Umherirrender) [08:29:44] (PS1) Umherirrender: Use foreach instead of ResultWrapper::fetchRow [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358909 [08:33:00] (PS2) Umherirrender: Make phpcs pass - includes [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358043 [08:33:17] (CR) Umherirrender: "Patch Set 2: Avoid conflict with I499ffbccd56dba56beb1ee96a502070c03f1d801" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358043 (owner: Umherirrender) [09:16:10] (CR) Thiemo Mättig (WMDE): [C: 2] Add phpcs and make pass [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358045 (owner: Umherirrender) [10:15:08] (CR) Umherirrender: "I resolve merge conflict when I915a6702a26721b8da82a59be6958d344543cefd gets merged, because both are changing lines near by each other w" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358050 (owner: Umherirrender) [12:36:13] Fundraising-Backlog, Continuous-Integration-Config, Gerrit, Release-Engineering-Team (Kanban): Fully retire MediaWiki "Donate" skin - https://phabricator.wikimedia.org/T124519#3347715 (hashar) I have: * marked the repository read-only in Gerrit * unconfigured it from CI * removed it from the meta... [12:36:43] Fundraising-Backlog, Continuous-Integration-Config, Gerrit, Release-Engineering-Team (Kanban): Fully retire MediaWiki "Donate" skin - https://phabricator.wikimedia.org/T124519#1958656 (hashar) a:hashar [14:29:18] (CR) Anomie: "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [15:06:20] (CR) Thiemo Mättig (WMDE): [C: 2] Make phpcs pass - includes [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358043 (owner: Umherirrender) [15:19:54] (Merged) jenkins-bot: Make phpcs pass - includes [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358043 (owner: Umherirrender) [15:46:33] (PS2) Umherirrender: Use foreach instead of ResultWrapper::fetchRow [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358909 [15:47:03] (PS4) Umherirrender: Add phpcs and make pass [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358045 [16:05:02] (PS4) Umherirrender: Use short array syntax - includes [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358050 [16:05:23] (CR) Umherirrender: "Patch Set 4: Rebased" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358050 (owner: Umherirrender) [16:15:37] fr-tech so last night AndyRussG and I discovered that any job which emits non-valid utf8 to stderr or std could hang the job runner such that it can't log the issue [16:15:45] fortunately the fix is really simple [16:15:59] just using the fallback option on the decode method [16:16:16] (PS4) Umherirrender: Use short array syntax [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/358052 [16:16:43] https://gerrit.wikimedia.org/r/358900 [16:16:56] we should probably get that deployed soon [16:19:37] ejegg: how did you fix it live? [16:19:51] unfortunately building the package takes a long time [16:20:23] cwd we stopped telling it to parse the wrong files [16:21:26] i guess the parser can handle bad utf8 ok in its own files [16:21:54] (PS2) Umherirrender: API: Split description messages into summary + additional text [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [16:21:59] (CR) Umherirrender: [C: 2] API: Split description messages into summary + additional text [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [16:22:04] but when the banner impressions parser script tries to read a landing page impression log file, which is ALSO bad utf, that's a world of hurt [16:22:23] interesting [16:22:34] because it writes to stderr and that is where p-c cries? [16:22:40] yepyep [16:23:01] makes sense [16:23:25] yeah i mean i won't push back on a release i'll just have to rearrange my schedule a little [16:23:45] so if we /do/ want another release, maybe we can get the autocompletion in there too! [16:23:54] for sure [16:24:02] i say we pile on whatever we're lacking [16:24:07] https://gerrit.wikimedia.org/r/351745 [16:24:08] long as it's low risk [16:24:23] (CR) jerkins-bot: [V: -1] API: Split description messages into summary + additional text [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [16:24:41] (Merged) jenkins-bot: API: Split description messages into summary + additional text [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [16:24:50] there are just 500 holes to fall in with the package build and i will fall in at least 5 of them each time [16:25:04] (CR) Umherirrender: [C: 2] "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358680 (owner: Anomie) [16:27:32] (CR) AndyRussG: [C: 2] "Woohoo! WFM :) Thx!" [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/358900 (https://phabricator.wikimedia.org/T167849) (owner: Ejegg) [16:28:38] cwd: thx much :) [16:28:43] ty AndyRussG [16:28:51] ejegg: likewise [16:29:15] So I guess we need to ask Jeff_Green about unicode issues further up the pipeline? [16:29:26] crazy that it's so complicated [16:29:47] IIRC analytics also had something to do with piping those requests to us, via a thing called kafkatee [16:29:47] AndyRussG: yeah, I guess we need to understand the difference between the new way the files are made and the old way [16:30:07] was kafkatee before or after udp2log? [16:30:13] after [16:30:47] (Merged) jenkins-bot: Handle invalid utf-8 coming from stderr/stdout streams [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/358900 (https://phabricator.wikimedia.org/T167849) (owner: Ejegg) [16:31:08] I guess there's a first step in Analytics's purview, then an intermediate step in FR-ops purview, then the Django Step in FR-Tech scope, and... finally the un-git-ized query code that FR-Online has made? [16:31:25] That's a lot of teams for one pipeline [16:31:48] whew! [16:32:17] (maybe the second step isn't a step, and I'm just imagining? we could at least ask analytics... They may know something about encoding there...) [16:32:38] heheh and also, ahead of analytics, is the Traffic team, which is in charge of the Varnishes that feed kafkatee [16:33:10] And then FR-Tech also does the client-side browser code that makes the request [16:33:26] Six steps anyway [16:33:51] from banner display to A/B test datum [16:34:45] if you guys have never read: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ [16:35:03] it is brutally complicated [16:37:00] Wikimedia-Fundraising-Campaigns (FY Q4 Apr-Jun 2017), Wikimedia-CentralNotice-Administration (Q2-2017): WMF Fundraising - 2016-17 - Poland/Romania/Slovakia (May 16 - Jun 13) - https://phabricator.wikimedia.org/T164268#3348756 (Pcoombe) Open>Resolved a:Pcoombe [16:37:33] ^ The main reason I started creating Phab tasks for campaigns is how satisfying it feels to close them [16:41:00] 'resolved' does come with a psychological boost [16:41:31] pcoombe: they should make a sound, like ker-chunk of somethng closing :) [16:43:48] cwd: nice! hmmm yes but ít ẃóŕḱŝ bētteŗ than old anglocentric ascii :) [16:44:12] heh yeah [16:44:38] after reading that article i stopped harboring any notion that i would one day grok character encoding [17:00:19] fr-tech: But the greatest Electrical Pioneer of them all was Thomas Edison, who [17:00:19] was a brilliant inventor despite the fact that he had little formal [17:00:19] education and lived in New Jersey. Edison's first major invention in [17:00:19] 1877, was the phonograph, which could soon be found in thousands of [17:00:19] American homes, where it basically sat until 1923, when the record was [17:00:19] invented. But Edison's greatest achievement came in 1879, when he [17:00:19] invented the electric company. Edison's design was a brilliant [17:00:20] adaptation of the simple electrical circuit: the electric company sends [17:00:21] electricity through a wire to a customer, then immediately gets the [17:00:21] electricity back through another wire, then (this is the brilliant [17:00:21] part) sends it right back to the customer again. [17:00:22] This means that an electric company can sell a customer the same batch [17:13:53] ejegg: all the banner impression log files known to have utf-8 errors, including the one where the error is on a line w/ a malformed beacon/impression call, are marked as ingested in the db [17:14:17] This makes sense, as the script does mark a file as done before ingesting: https://github.com/wikimedia/wikimedia-fundraising-tools-DjangoBannerStats/blob/master/fundraiser/analytics/management/commands/LoadBannerImpressions2Aggregate.py#L107 [17:14:57] (squidlog is the ingested files table and SquidLog is the corresponding Python class) [17:15:22] I guess that could be precisely for circumstances like this, so that it doesn't get stuck on bad files, but just keeps going [17:17:20] Anyway, I think the historical truth () here is that it was bad luck that a line for a malformed beacon/impression call had malformed utf-8, too. Sound right (to write up in the task)? [17:17:53] http://www.jornada.unam.mx/ultimas/2017/03/17/reitera-gobierno-la-2018verdad-historica2019-sobre-los-43-ante-cidh [17:33:45] Fundraising Sprint Kickstopper, Fundraising-Backlog, Unplanned-Sprint-Work: Banner impression loader failing on malformed calls to beacon/impressions - https://phabricator.wikimedia.org/T167557#3348886 (AndyRussG) So it turns out this was a bit more complex than we thought... @Ejegg and I found that... [17:33:51] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Update 'Amazon Payments' branding - https://phabricator.wikimedia.org/T158016#3348888 (Pcoombe) Open>Resolved Okay, I've updated: - Copy on [[ https://wikimediafoundation.org/wiki/Ways_to_give | Ways to Give ]] and the [[ https://meta.wikime... [17:35:48] any scrum of scrums news fr-tech? [17:36:05] ejegg: just the BS deploy! :) [17:36:13] happened Monday [17:36:19] cool! [17:36:33] nothing here, ty [17:36:41] And re: the earler question, yeah, sounds like exactly what must have happened [17:38:54] ejegg: cool, thx... pls go ahead and add/correct anything on the task as needed :) [17:45:18] (PS2) Ejegg: Script to update d*Local payment submethods [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/358154 (https://phabricator.wikimedia.org/T167378) [17:51:54] (PS1) Ejegg: WIP / DO NOT MERGE: d*local methods from API [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/359003 (https://phabricator.wikimedia.org/T167378) [17:53:19] (PS2) Ejegg: Batch look up order status for Ingenico [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/328858 [17:55:46] XenoRyet|afk: I feel like we should try to get mepps familiar with paypal EC [19:15:40] (PS1) Ejegg: Config fix: Check provider config for actions node [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/359020 [19:26:56] ejegg do you have a minute for a quick chat? [19:27:13] sure! [19:27:39] (CR) Mepps: [C: 2] Config fix: Check provider config for actions node [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/359020 (owner: Ejegg) [19:27:58] okay i'm in fr-tech-talk [19:30:24] (Merged) jenkins-bot: Config fix: Check provider config for actions node [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/359020 (owner: Ejegg) [19:44:56] ejegg do you get 6 errors and 2 failures if you run the wmf_civicrm tests? [19:45:13] I'll check [19:45:23] I think it was just one failure [19:45:48] on the refund amount [19:45:50] last time [20:32:36] Fundraising Sprint Kickstopper, Fundraising-Backlog, FR-PayPal-ExpressCheckout: PayPal EC dead session error - https://phabricator.wikimedia.org/T167923#3349710 (XenoRyet) [20:35:40] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: De-duplicate result switcher code - https://phabricator.wikimedia.org/T112839#3349725 (Ejegg) Open>Resolved p:Triage>Normal a:cwdent [22:10:06] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1446 [22:15:06] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1745 [22:20:06] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 2624988 Threads: 1 Questions: 50682731 Slow queries: 14343 Opens: 13683 Flush tables: 1 Open tables: 607 Queries per second avg: 19.307 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 598 [22:22:18] ^ ? [22:24:45] AndyRussG: as far as i can tell it is network flap to codfw [22:27:26] Huh [22:27:49] We have stuff in both places? [22:35:09] yeah we have an almost working parallel cluster over yonder [22:35:22] anything that says fr*2* is codfw [22:35:46] as well as anything named after a star (vs an element) [22:38:34] cwd: ah nice! [22:38:51] So the idea is to be able to switch back and forth transparently [22:38:54] ? [22:39:02] Ah right ^ I see it says frdb2001 [22:39:12] AndyRussG: that would be ideal :) [22:39:29] we did it once but have changed a few key pieces of infrastructure in eqiad since then [22:40:21] and for some reason i have been seeing a lot of network lag lately. we have our own set of fundraising firewalls in both places so that could be related [22:41:00] we auto rsync some giant stuff around so it could be saturating an interface or something [22:48:15] hmm interesting [22:48:17] cool! [22:48:26] coolsteresting [22:54:42] gotta run! [23:42:01] (PS1) Ejegg: Revert "Allow specifying configuration dir with environment vars" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/359078 [23:42:03] (PS1) Ejegg: WIP get provider config dir from global config [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/359079 (https://phabricator.wikimedia.org/T147503)