[00:00:29] hmm - not quite sure - because the one Leannes sent me wasn't like that [00:00:37] now I've confused myself.... [00:00:49] however, I think that minimum fix will be OK for now [00:01:24] ok so you want to take out the error message? [00:02:28] yep - that's my thinking [00:02:49] my brain has malfunctioned - I can no longer replicate the error - but I'm sure I demo'd it to you [00:06:34] OK - it only happens on address not email - that's why [00:08:22] (PS1) Awight: Try upload 3 times [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/325446 (https://phabricator.wikimedia.org/T152352) [00:09:39] (PS2) Eileen: Do not give validation error on address with no location_type. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/325444 (https://phabricator.wikimedia.org/T152426) [00:11:17] fr-tech: Quick silverpop export review... //gerrit.wikimedia.org/r/325446 [00:11:20] uh [00:11:22] https://gerrit.wikimedia.org/r/325446 [00:12:52] (PS3) Eileen: Do not give validation error on address with no location_type. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/325444 (https://phabricator.wikimedia.org/T152426) [00:13:17] If anyone has time to review that I have simply commented out one line of form submission validation that was causing problems [00:13:22] should be easy to review [00:13:39] It would be good to fix the data but better to have more time on that [00:13:54] (I'm shooting out for lunch now) [00:14:03] !nick eileen[afk] [00:14:18] (CR) Awight: [C: 2] "O_o. Hide the knives!" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/325444 (https://phabricator.wikimedia.org/T152426) (owner: Eileen) [00:16:24] thanks awight [00:17:40] awight: jenkins retry logic not up to snuff? [00:18:15] hmm, seems my vagrant patch needs fixin' [00:19:38] the vagrant patches, they are a-fixin' [00:19:41] (Merged) jenkins-bot: Do not give validation error on address with no location_type. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/325444 (https://phabricator.wikimedia.org/T152426) (owner: Eileen) [00:19:53] eileen[afk]: your current bandaid sounds good. thanks! [00:19:57] ejegg: I didn't see any retry logic [00:20:35] ejegg: hey... when u have a sec, I'm still questionful about exactly you pulled info on datacenters for banner dip investigation [00:21:05] I see the query u put in the etherpad with "GROUP BY x_cache, hour;" [00:21:28] ah, oops, I must have pasted the wrong thing in there AndyRussG [00:21:33] Ah K [00:21:35] thx! [00:21:43] Should be grouping on a substr of the hostname [00:21:52] Yeah it seems x_cache itself is more complex [00:21:54] Ah k [00:21:58] hmmm [00:23:10] AndyRussG: datacenter query looks right on line 262 of the etherpad [00:23:21] ...GROUP BY substr(hostname, 8), SUBSTR(ts,1,16); [00:24:10] Ah hmmm [00:25:38] ejegg: do you know what that means exactly in the chain of hosts that the request may go thru? [00:25:50] https://wikitech.wikimedia.org/wiki/Varnish#X-Cache [00:26:52] Does it necessarily tell us which DC the request went to first? [00:27:01] (CR) Awight: [C: 2] "Sort of works locally ;)" (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/324343 (https://phabricator.wikimedia.org/T151954) (owner: Ejegg) [00:27:46] AndyRussG: shoot, no, I don't [00:28:55] (Merged) jenkins-bot: Less noisy queue consumer logging [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/324343 (https://phabricator.wikimedia.org/T151954) (owner: Ejegg) [00:32:12] ejegg: So, did you see something about retry in Jenkins? I imagine there's a plugin, but didn't find any controls for that. [00:32:54] nah, I assumed you knew about one [00:33:04] reviewing... [00:33:33] ty! [00:35:52] ejegg: ah K thx! [00:36:41] (CR) Ejegg: [C: 1] "Looks good, tiny error formatting thing" (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/325446 (https://phabricator.wikimedia.org/T152352) (owner: Awight) [00:36:43] (CR) Awight: [C: 2] Update thank you letter. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/323988 (https://phabricator.wikimedia.org/T151814) (owner: Eileen) [00:37:13] (PS2) Awight: Try upload 3 times [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/325446 (https://phabricator.wikimedia.org/T152352) [00:37:57] (CR) Ejegg: [C: 2] Try upload 3 times [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/325446 (https://phabricator.wikimedia.org/T152352) (owner: Awight) [00:40:47] (Merged) jenkins-bot: Update thank you letter. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/323988 (https://phabricator.wikimedia.org/T151814) (owner: Eileen) [00:40:49] (Merged) jenkins-bot: Try upload 3 times [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/325446 (https://phabricator.wikimedia.org/T152352) (owner: Awight) [00:42:49] (PS1) Awight: Merge master into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/325449 [00:44:29] (CR) Awight: [C: 2] Merge master into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/325449 (owner: Awight) [00:46:29] ejegg: That V-2 on your vagrant patch is insane. It's a single-quote lint warning on a file the patch doesn't touch. [00:47:37] yeah, i had the same problem with my last vagrant patch [00:47:51] i dunno how these lint-failing things keep merging [00:48:12] (Merged) jenkins-bot: Merge master into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/325449 (owner: Awight) [00:49:16] !log update fundraising-tools from 43982478d248cfe99511328dec08dccd6105d4b6 to 931c8cf1db720723f90307f6a417184bd8c03bfc [00:49:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:51:02] (CR) Awight: [C: 2] Upgrade to Drupal 7.52 upstream [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/322822 (owner: Eileen) [00:51:16] (CR) Awight: [V: 2] Upgrade to Drupal 7.52 upstream [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/322822 (owner: Eileen) [00:51:54] (PS1) Eileen: Submodule update. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325451 [00:54:05] (CR) jenkins-bot: [V: -1] Submodule update. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325451 (owner: Eileen) [00:57:01] (Merged) jenkins-bot: Re-apply WMF patches [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/322823 (owner: Eileen) [00:57:50] (CR) Awight: [C: 2] Simulate 1000 emails (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/322383 (owner: Ejegg) [01:01:14] (Merged) jenkins-bot: Simulate 1000 emails [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/322383 (owner: Ejegg) [01:24:08] AndyRussG: I tried grouping by a whole host of things, including ipv6 (using client_ip like '%:%') [01:24:34] and the project language is still the only big determinant [01:34:11] (CR) Eileen: [C: 2] "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325451 (owner: Eileen) [01:36:27] ejegg: hmmm K... [01:37:30] (Merged) jenkins-bot: Submodule update. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325451 (owner: Eileen) [01:41:06] (PS1) Eileen: Alternate fix for reconciliation report doubling some rows. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325463 (https://phabricator.wikimedia.org/T152238) [01:41:10] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325464 [01:42:11] (Abandoned) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325464 (owner: Eileen) [01:42:30] (Abandoned) Eileen: Alternate fix for reconciliation report doubling some rows. [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325463 (https://phabricator.wikimedia.org/T152238) (owner: Eileen) [01:44:43] (PS1) Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325465 [01:45:02] (CR) Eileen: [C: 2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325465 (owner: Eileen) [01:45:10] (Merged) jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325465 (owner: Eileen) [01:46:20] ooh, weird! en/US/Chrome Mobile 38/ipv4 actually got MORE impressions during the dip, while en/everything else was down by like 97% [01:47:19] ejegg: Are you talking about the Dec 2 dip or did a new dip happen? [01:47:24] the Dec 2 dip [01:47:32] Hmm OK [01:47:59] So we lost 95% of our impressions but somehow Chrome 38 on mobile went up [01:48:20] What %age of our paltry banner traffic was Chrome 38 on mobile then? [01:50:37] pasted my query on https://etherpad.wikimedia.org/p/T152122_notes [01:52:39] RoanKattouw: from 9:10-9:55 en/US/chrome mobile/v4 had 14k banners on 148k page views [01:53:00] but from 8:10-8:55 had 16k banners on 130k page views [01:53:26] hah [01:53:34] * RoanKattouw doesn't have access to the DB you're running those queries against [01:53:50] And what was the total # of banners served during that time period? [01:54:16] Probably at least 45k since I recall #banners/sec being around 900-1000 during that time [01:54:46] So that would suggest that Chrome mobile was 1/3 of our banner traffic during the dip [01:55:09] total en/US was 23k [01:55:32] specifically chrome mobile 38, on ipv4 [01:55:43] hah so 2/3? [01:56:37] for en/US, yeah [01:57:02] so... wtf is different with chrome mobile 38? [01:58:51] !log CiviCRM update from 0f1e80b0c4cb97a7972cfd9f880fd36ab0b5d8c3 to 36a49c5f0bede085ecc86de310be6e41468ac045 [01:59:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:59:27] Maybe it has buggy caching behavior? [01:59:38] I was thinking about this earlier, about why impressions didn't go all the way down to 0 [02:00:08] And theorized that maybe some browsers cached choiceData too aggressively and so never updated their cached choiceData during that hour [02:00:22] So I expected that 5% to be old IE versions or something [02:00:34] But since it's mobile, I wonderif it's all from one telco's IP range [02:00:53] Maybe there's a telco with a caching proxy in front of us that's misbehaving [02:01:02] Although that seems unlikely since our domains are all HTTPS-only [02:02:47] ahhh, samsungBrowser shows up as Chrome Mobile 38 [02:04:57] hmm [02:06:55] hmm indeed [02:07:04] explains why 38 is so popular for such an old one, anyway [02:07:40] ejegg: that's the last query on the etherpad? [02:07:54] yep [02:07:58] RoanKattouw: This is Hive webrequests, u can get it on stat1002 [02:08:11] I don't have access to stat1002 [02:08:13] I exported to a csv and have been sorting by different columns [02:08:37] There's just enough bureaucracy around getting it and I have little enough reason to have it that I've never requested it [02:09:04] aww [02:09:09] it's fun [02:09:29] ejegg: looks like quite a mighty query [02:10:01] RoanKattouw: pretty sure the CSV is aggregated enough I can email it around [02:10:36] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Add option to early-exit dedupe jobs based on the volume of contributions being processed - https://phabricator.wikimedia.org/T152072#2849589 (Eileenmcnaughton) Open>Resolved [02:10:37] only combos with > 100 page views in both hrs, and > 100 impressions outside of the bug time [02:12:09] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Fix data where contacts have multiple emails or addresses of the same location type (or have no location type) - https://phabricator.wikimedia.org/T152474#2849591 (Eileenmcnaughton) [02:12:23] I'm mostly just idly curious about whether the dip-time traffic is only a few browsers [02:12:33] It sounds like Chrome 38 Mobile accounts for 2/3 of it in the US [02:12:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Fix data where contacts have multiple emails or addresses of the same location type (or have no location type) - https://phabricator.wikimedia.org/T152474#2849604 (Eileenmcnaughton) [02:13:28] ejegg: pls do send! :) [02:14:34] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Figure out why new recurring entries are creating duplicate emails of the same location type, ditto address - https://phabricator.wikimedia.org/T152475#2849619 (Eileenmcnaughton) [02:16:30] ejegg: the dip doesn't seem to end until after 9:20... I'm using 7 and 10 hrs as baseline... [02:16:42] (both Dec. 1 and 2) [02:17:16] oh really? [02:17:38] Hmm so the most common banner recipient was IE 11 in Germany; but did you say that de was an unaffected language [02:17:39] ? [02:18:00] RoanKattouw: I think de was affected, nl wasn't? [02:18:07] Hmm [02:18:25] Oh whoops I should have sorted by banners_dip instead [02:19:00] So make that iOS 10 in JApan [02:19:15] ejegg: is this right? ( ts > '2016-12-02 08:10' AND ts < '2016-12-02 08:55' ) OR [02:19:17] ( ts > '2016-12-02 09:10' AND ts < '2016-12-02 09:55' ) [02:19:21] But yeah WTF @ the dip percentage of -28.7% for Chrome 38 on Android [02:19:33] Did u mean on for Dec 1 and another for Dec 2 [02:19:41] ejegg: don't mean to criticize, this looks great!! [02:19:48] AndyRussG: I wanted to get the same chunk of each hour, and I thought that 8:10-8:55 was firmly in the outage period [02:21:16] ejegg: Ah K I see, so firmly within or without outage period, excluding transitions, all for Dec 2? [02:21:26] AndyRussG: yep! [02:22:07] so pageviews, banners, and ratio are from 9:10-9:55, and the _dip variations are from 8:10-8:55 [02:22:22] K [02:23:48] Now where can I put a query in the etherpad? [02:26:28] heh, oops - right above mine! [02:28:15] np, yeah working notes just above, labeled "Rought notes for querisilliness (AG):" [02:34:58] checking to see whether basically all the chrome mobile 38 requests are actually the hacked up samsung version [02:35:30] hmmm [02:35:35] exploding samsung js [02:38:15] it must be something like what RoanKattouw ^ said, bad caching, no? [02:39:06] Haha [02:39:17] (@ exploding Samsung [02:39:53] Well I mean I'm still assuming the cache poisoning theory is correct so that colors my views I think [02:40:33] Mmm I think this is evidence for it [02:41:03] As is the language variance, since it's available server-side, choicedata varies on language [02:41:18] and the 1-hr duration of the outages [02:41:25] Don't tell anyone but I'm hoping we'll get another dip so our monitoring will actually be able to log interesting things [02:41:34] Ooh, hmm [02:41:52] ? [02:42:15] Do we have cross tabs by any chance to see if say Norwegian-speaking users on enwiki were affected? [02:42:25] BTW I did ask AaaronSchulz [02:42:54] if he could think of any possible cache-related explanations [02:43:30] I.e. the wiki language is an affected language but the user language is not, or vv [02:43:39] RoanKattouw: or maybe vice versa? that would make more sense [02:43:59] Is the choiceData cache by wiki or by user lang or both? [02:44:30] (RL URLs contain both) [02:45:07] But I guess 1h points to the CD cache, there are no 1h RL caches [02:46:23] RoanKattouw: enwiki users in Norway were affected [02:46:52] ah, but I didn't snag the accept-language header [02:47:03] Or even their user preference [02:47:22] Whatever determines which lang they get banners in [02:47:27] ehh, >99% anon [02:54:02] RoanKattouw: or more specifically, whatever value makes it to the lang param in RL context [03:23:33] Right [03:23:45] Oh, if it's >99% anon then wiki lang === user lang [03:28:03] hmm, about 50% of chrome mobile 38 has user_agent like '%Samsung%' wonder if there's a correlation to banner impressions [03:41:56] huh, almost all 16k of the chrome mobile 38 views during the dip were Android 4.2.1 on Nexus 5 [03:47:00] wtf [03:47:10] I have a Nexus 5, and I'm pretty sure it's running Android 6 [03:47:22] Maybe these are Nexus 5s sold by someone other than Google? [03:49:29] wondering if it's one malfunctioning device... [03:50:20] grouping by first few chars of IP address [03:53:00] well, 1 IP address is like 7.5k of them. just a mobile NAT? [03:53:13] wow crazy [03:53:25] look up the IP address? [04:01:25] So maybe it's a crawler? [04:01:30] that got stuck? [04:02:56] There's this stuff: https://developer.chrome.com/multidevice/data-compression-for-isps [04:03:17] but is it bypassing https somehow? [04:07:18] ejegg: we could check for a difference between http and https connections... Hmmm don't see it, maybe in x_analytics_map? [04:08:37] yeah, I don't see it either [04:09:00] checking the referer [04:10:01] ejegg: https://wikitech.wikimedia.org/wiki/X-Analytics it's there! [04:10:13] Also a bunch of other stuffs, such as logged-in-nes [04:10:15] ness [04:11:02] Ah hmmm maybe https/http is only for mobile? [04:11:29] are we running the mobile campaign yet? I thought I understood from standup this morning that not yet. Or maybe it's just low-level... [04:12:25] yeah, i think it's the low level worldwide [04:14:30] ooh, weird. referer is all https, but with a heavy slant to topics from India. Let's check that zero header! [04:16:21] Well it was also midnight - 1am PST, 3am-4am PST, and 1:30pm-2:30pm IST [04:16:36] So I think there would be some slant towards India regardless [04:16:47] (Though it was also 8am-9am GMT...) [04:19:15] like, super heavy slant - 19 of the top 20, and the last is the article for Gmail [04:20:15] some India mobile company MITM-ing all their phones' ssl requests to send 'em through compression proxies? [04:20:49] anyway, proxy fits with your cache poisoning idea [04:22:34] oh right, we don't have zero in India [04:33:51] huh, vodafone [04:34:19] well, this is fun, but doesn't tell us anything about why the rest of the traffic dried up [04:34:39] signing off for the night... [09:21:05] How many times should a user see a banner? [09:21:15] [09:20:44] yeah but it keeps coming back and it's big and red and scary [09:22:03] Or, relatedly, how many times should they have to dismiss it to make it go away completely [11:17:54] Reedy, dismissed twice within a single browser. One big one small [17:50:47] Fundraising-Backlog, FR-Paypal: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851138 (awight) [17:52:35] dstrine-away: ^ [17:53:04] awight: thanks. heading in... [17:53:55] ill take a look at that [17:53:56] my mess [17:55:20] fr-tech currently exporting to CSV the query in the etherpad, below where it says "Rought notes for querisilliness (AG)"... https://etherpad.wikimedia.org/p/T152122_notes [17:56:54] If it works I'm gonna try grouping on permutations of lang + DC [18:00:02] fr-tech: When I think about myself, [18:00:02] I almost laugh myself to death, [18:00:02] My life has been one great big joke, Sixty years in these folks' world [18:00:02] A dance that's walked The child I works for calls me girl [18:00:03] A song that's spoke, I say "Yes ma'am" for working's sake. [18:00:03] I laugh so hard I almost choke Too proud to bend [18:00:03] When I think about myself. Too poor to break, [18:00:04] I laugh until my stomach ache, [18:00:04] When I think about myself. [18:00:05] My folks can make me split my side, [18:00:05] I laughed so hard I nearly died, [18:00:06] The tales they tell, sound just like lying, [18:00:06] They grow the fruit, [18:00:07] But eat the rind, [18:00:54] awight: is what's weird with that the name in the middle of the address? [18:01:41] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Temporarily turn off required "postmark date" until Jan 1 - https://phabricator.wikimedia.org/T152512#2851178 (awight) [18:01:50] cwd: yep [18:02:23] supplemental_address_1? [18:03:04] that sounds like what happened [18:03:06] weirdness [18:03:35] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Temporarily turn off required "postmark date" until Jan 1 - https://phabricator.wikimedia.org/T152512#2851200 (awight) p:Triage>High [18:07:50] pretty sure that was copied from the previous listener [18:08:48] i have to gather some fire wood, it is freezing in here! [18:12:24] cwd|afk: the source_* metadata will tell [18:14:36] says it's the new listener, commit 36be698face889cef0be54d9388259e5617e0987 [18:14:44] what you mean by "copied"? [18:14:47] oooh [18:14:48] the code. [18:14:49] right [18:16:04] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: investigate the ingenico reintegration - https://phabricator.wikimedia.org/T149238#2851236 (Ejegg) Good news: Connect appears to be almost the exact same API calls, just using REST instead of XML... [18:17:03] fr-tech: droogs, I wanted to draw attention to WMDE's fundraising code rewrite [18:17:10] I forwarded an email... [18:17:25] Yea, I saw that, but hadn't had a chance to really look thought it yeet. [18:17:33] oho! opportunity to collaborate? [18:17:37] oops [18:17:47] yah [18:17:55] I haven't gone through their code yet, either [18:18:00] but it's bound to have some gems [18:19:04] nice [18:19:11] join forces [18:23:41] Fundraising-Backlog, FR-Paypal: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851138 (cwdent) We map their field address_name to our field supplemental_address_1. Is there a better place to assign that value? [18:29:43] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Unplanned-Sprint-Work: Central Notice: possible CN issue early on December 1st UTC - https://phabricator.wikimedia.org/T152122#2851304 (Ejegg) Oh hey, 08:00 UTC is 00:00 PST. Midnight cron job, or witch... [18:31:08] (PS1) Awight: Only enforce postmark date rule during January [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325597 (https://phabricator.wikimedia.org/T152512) [18:34:10] (CR) Ejegg: [C: -1] "Ahh, you're right, a WHERE clause makes more sense. Might as well do inner joins in that case, right? -1 ing for total original currency " [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325250 (https://phabricator.wikimedia.org/T152238) (owner: Eileen) [18:37:31] Fundraising-Backlog, FR-Paypal: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851344 (awight) Ah. I just checked with @CaitVirtue, and my understanding of the problem was wrong. Yeah, all that happened is that the name is appearing in two places which is sil... [18:37:55] Fundraising-Backlog, FR-Paypal: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851348 (awight) [18:40:11] (CR) Ejegg: Only enforce postmark date rule during January (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325597 (https://phabricator.wikimedia.org/T152512) (owner: Awight) [18:46:40] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851395 (CaitVirtue) [18:48:18] (PS2) Awight: Only enforce postmark date rule during January [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325597 (https://phabricator.wikimedia.org/T152512) [18:48:46] (CR) Awight: Only enforce postmark date rule during January (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325597 (https://phabricator.wikimedia.org/T152512) (owner: Awight) [18:53:22] (CR) Ejegg: [C: 2] Only enforce postmark date rule during January [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325597 (https://phabricator.wikimedia.org/T152512) (owner: Awight) [18:57:44] (Merged) jenkins-bot: Only enforce postmark date rule during January [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325597 (https://phabricator.wikimedia.org/T152512) (owner: Awight) [19:02:39] (PS1) Awight: Merge master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325601 [19:02:49] (CR) Awight: [C: 2] Merge master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325601 (owner: Awight) [19:07:37] (Merged) jenkins-bot: Merge master into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325601 (owner: Awight) [19:08:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851459 (DStrine) @awight just ran a report and we got 81,000 records from the last 2 days. We need to plug the hole, wherever it is. [19:08:33] !log update civicrm from 36a49c5f0bede085ecc86de310be6e41468ac045 to d20c1c49ff99bdeb47dc183482ecc29abbc86280 [19:08:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:08:45] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851460 (DStrine) p:Triage>Unbreak! [19:09:35] dstrine-away: i am working on it, but it's not new behavior [19:10:02] i'm pretty sure it's people configuring their paypal account with their name in the business name section [19:10:12] +1 [19:10:17] that makes sense [19:11:18] (PS1) XenoRyet: Adyen form for Canada [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/325604 (https://phabricator.wikimedia.org/T152122) [19:12:55] (PS1) Cdentinger: reject some address_names [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) [19:16:42] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Figure out why new recurring entries are creating duplicate emails of the same location type, ditto address - https://phabricator.wikimedia.org/T152475#2851510 (DStrine) @LeanneS and @Rosiestep do you have any contact ID where you see this? I would lik... [19:19:27] (CR) Awight: reject some address_names (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) (owner: Cdentinger) [19:21:03] (CR) Awight: Adyen form for Canada (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/325604 (https://phabricator.wikimedia.org/T152122) (owner: XenoRyet) [19:22:06] ccogdill: How does the silverpop import look from your side? [19:22:28] excellent! processed twice last night (because we had to catch up on the 12/5 file [19:22:33] and then 12/6 came through with a few more rows [19:22:39] perfectly [19:22:55] oh great! [19:23:35] fyi, the upload finished at 09:54 UTC [19:25:09] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, FR-Email, Patch-For-Review, Unplanned-Sprint-Work: Silverpop export: retry FTP upload on failure - https://phabricator.wikimedia.org/T152352#2851547 (awight) Open>Resolved New split export/upload is confirmed working, we'll se... [19:25:24] (CR) Cdentinger: reject some address_names (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) (owner: Cdentinger) [19:26:01] (CR) Awight: reject some address_names (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) (owner: Cdentinger) [19:28:42] (PS2) Cdentinger: reject some address_names [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) [19:29:33] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, Patch-For-Review: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851581 (cwdent) I believe this is due to people configuring their paypal profile with their name in the "business name" part [19:29:37] (PS2) XenoRyet: Adyen form for Canada [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/325604 (https://phabricator.wikimedia.org/T152123) [19:30:35] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review, Unplanned-Sprint-Work: Central Notice: possible CN issue early on December 1st UTC - https://phabricator.wikimedia.org/T152122#2839139 (XenoRyet) Ignore above. Wrong bug number in... [19:33:45] (CR) Eileen: "OK - I can ditch the original currency - as an optional field it does mean that if you enable it you can validate the results more easily " [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325250 (https://phabricator.wikimedia.org/T152238) (owner: Eileen) [19:36:30] (CR) jenkins-bot: [V: -1] reject some address_names [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) (owner: Cdentinger) [19:41:02] (PS3) Cdentinger: reject some address_names [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) [19:46:20] can anyone think of a time when there wouldn't be a correspondence between wgNoticeProject and wgDBname? [20:01:26] BTW looks like datacenter is definitely not related to the CN outage. Looking at grouping on language and dc, seems that the differences in dc is just due to very sparse data for the languages with the outage at some dc's [20:07:21] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, Patch-For-Review: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851922 (DStrine) p:Unbreak!>High [20:31:00] https://twitter.com/aaronshaw/status/806232137141780480 [20:35:53] (CR) Awight: [C: 2] reject some address_names [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) (owner: Cdentinger) [20:36:10] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Figure out why new recurring entries are creating duplicate emails of the same location type, ditto address - https://phabricator.wikimedia.org/T152475#2852123 (Eileenmcnaughton) This was something I picked up in DB queries - I'll find some examples [20:36:39] fr-tech: bingo!! [20:36:50] The CN outage is definitely a choicedata issue [20:37:09] Checking outages on combinations of lang + project [20:37:13] goodaverage badaverage ratio uselang db [20:37:15] 15657.5416666667 135.431818181818 0.008649622084043 de dewiki [20:37:17] 243.058333333333 2.86111111111111 0.011771294042354 fr frwiki [20:37:21] 31957.9666666667 656.340909090909 0.020537630442412 en enwiki [20:37:22] 1567.35833333333 41.4090909090909 0.026419670619306 it itwiki [20:37:23] 39.0333333333333 1.65625 0.042431682322801 en simplewiki [20:37:25] 6.05042016806723 4.31818181818182 0.713699494949495 he hewiki [20:37:27] 1.34782608695652 1 0.741935483870968 es eswikiversity [20:37:29] 1.21739130434783 1 0.821428571428571 fr frwikisource [20:37:53] Only 4 combinations account for the totality of the outage (on Dec. 1) [20:38:03] Sorry 5 [20:38:19] de on dewiki, fr on frwiki, en on enwiki, it on itwiki and en on simplewiki [20:42:43] Ah hmmm ^ K I take it back :( [20:42:49] Not totally bingo [20:43:44] I don't see any data for these languages on any other projects that is significant enough to be worthy of consideration [20:44:14] (Merged) jenkins-bot: reject some address_names [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/325607 (https://phabricator.wikimedia.org/T152510) (owner: Cdentinger) [20:45:08] Hmmm [20:45:24] K ^ I think we'd need to munge the data better to be sure [20:46:04] Oh well... Probably to continue on this we'd just want to go with the assumption it was a poisoned ChoiceData cache and look for the cause... [20:46:16] nothing seems to indicate that it isn't [20:52:40] AndyRussG: are you still looking at: https://phabricator.wikimedia.org/T152122 [20:53:17] I thought we were pausing that. It sounded like a rabbit hole until the issue re-emerges. I'd like to take it out of the sprint [20:56:35] dstrine: Speaking of things we might want to take out of sprint. I'm still not done with the Canada thing. The form works, but the thing that keeps bogging me down is there's damn near nine hundred thousand Canadian postal codes. Getting all that data massaged into the right format to go into civi is proving logistically difficult in silly ways. I am close, it's just everything takes longer than I expect on this one. [20:56:50] Should I keep plugging away, or set it aside and start that investigation? [20:56:52] (CR) Awight: Add gitignore and tox support (4 comments) [wikimedia/fundraising/stats] - https://gerrit.wikimedia.org/r/324774 (owner: Awight) [20:57:10] dstrine: just following up on some queries I'd started on... yeah all evidence points to a specific rabbit hole... likely in the same carrot patch as the previous missing banners bug [20:57:24] * XenoRyet always forgets to turn of his lunch notice [20:57:40] i.e. due to a failure in more or less the same subsystem of Mediawiki core [20:57:58] XenoRyet: can you just put back the province/city fields? [20:58:09] Yea, that's cake. Take two minutes. [20:58:23] Do that and move on? [20:58:24] dstrine: I'd just summarize on the task where we got so far, and then leave it, I guess... [20:58:41] XenoRyet: cool yeah do that and move on [20:59:19] dstrine: what should I work on? Tying up T149107? [20:59:19] T149107: CentralNotice: Relay banner loading issues in beacon/impression - https://phabricator.wikimedia.org/T149107 [20:59:44] AndyRussG: ok cool. Slightly related, there was a change that didn't go out last week due to risk. Do we have to hold it this week too? [20:59:47] I think someone from fr-tech should at least look over the cps reports, too [20:59:56] Jeff_Green: since there is no op in this channel, is there any way to kick someone? [21:00:01] dstrine: yes, already checked, it's still being held back [21:00:26] the matrix people say those cwd* bots will stay gone and be gone [21:00:44] ok, great, thanks AndyRussG [21:05:38] (PS3) XenoRyet: Adyen form for Canada [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/325604 (https://phabricator.wikimedia.org/T152123) [21:05:44] cwd ehwut? [21:05:54] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, FR-Ingenico, Unplanned-Sprint-Work: Spike: investigate the ingenico reintegration - https://phabricator.wikimedia.org/T149238#2852265 (Ejegg) After we configure the look and feel of the hosted checkout page, migration of credit card donat... [21:07:00] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, FR-Adyen, Patch-For-Review, Unplanned-Sprint-Work: Adyen: setup a form for Canada - https://phabricator.wikimedia.org/T152123#2839215 (XenoRyet) Due to geocoding difficulties, we're keeping the city and province fields for now. [21:15:36] cwd: Want any help with the dash-vagrant work? [21:16:52] awight: i have hit a wall with a provision error [21:23:36] (PS2) Ejegg: Alternate fix for reconciliation report doubling some rows. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325250 (https://phabricator.wikimedia.org/T152238) (owner: Eileen) [21:25:28] (CR) Ejegg: [C: 2] "Sorry, I withdraw my objections about the original_currency! I thought the statistics key meant it was going in the grand total line, but " [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325250 (https://phabricator.wikimedia.org/T152238) (owner: Eileen) [21:25:44] ejegg: do you think that might have anything to do with the 11/30 discrepancy? [21:27:31] woo! [21:27:37] kick cwd[m]1 [21:28:35] cwd maybe! Eileen pointed out that we'd added the restriction on a table that was being left joined, so the rows stuck around [21:28:47] I'mma deploy and see if it fixes the report [21:29:37] hooray! thanks Jeff_Green [21:29:44] ejegg: fingers crossed! [21:29:47] cwd: no problem. [21:32:03] fr-tech: anyone care if i deploy smashpig? here's what's in the hopper: [21:32:06] + 5db2b2109071c9d642aefb09d7d15375f918551f Clean up a couple unused things [21:32:09] + 78a59db62eccfbc6cfa9ab08443e4cba31517ca4 Update payments_initial when donation completes [21:32:11] + e181e464087da17759f34b845fe7bd3995662e0b Less noisy queue consumer logging [21:32:13] + bbba45f67287ea94afd4eec5a982685ff33e23be reject some address_names [21:32:38] cwd ooh, that payments_initial one will need a settings update [21:33:56] ejegg: could these be destabilizing? the address_name thing is just populating a field we don't really need, that data can get cleaned up whenever [21:34:21] cwd: cool, want to clue me into the provisioning error? Should I pull your patch or anything? [21:34:40] cwd: wanna just cherry-pick the PP listener fix [21:34:52] sure that's fine [21:34:53] maybe everything except the payments_initial perhaps [21:34:58] (CR) Paladox: "check experimental" [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/277768 (owner: Paladox) [21:35:02] (CR) Paladox: "recheck" [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/277768 (owner: Paladox) [21:36:09] cwd: k I see. "WIP add dash to crm role" [21:37:29] cwd could do the queue consumer logging too [21:40:19] urgh, there's a conflict with a comment that says livehack [21:40:22] Jeff_Green: good news! It looks like the node ppl DO mess around with apt: https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions [21:40:54] ejegg cool, that makes things easier [21:41:17] i was just noticing that even stable deb has a pretty new node [21:41:23] (Merged) jenkins-bot: Alternate fix for reconciliation report doubling some rows. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/325250 (https://phabricator.wikimedia.org/T152238) (owner: Eileen) [21:41:27] I think that thing they say to curl | sudo just adds their repo [21:41:33] but i think that's a recent change cause i remember it being like 0.10 not that long ago [21:42:59] everything about that curl step makes me weep [21:43:31] trust us [21:43:44] we promise we aren't bad. here have a cookie [21:44:33] we should turn dash into an appliance [21:47:43] yep, script just adds their signing key and their repo, and runs apt-get update [21:48:10] yah the script itself looks sane [21:51:24] cwd|brb: I have a different error. ==> default: Notice: /Stage[main]/Crm::Dash/Git::Clone[dash]/Exec[git_clone_dash]/returns: Host key verification failed. [21:52:13] awight: interesting, that's github [21:53:13] i need to run my vm on a vm because vagrant up == verizon down [21:53:31] which reminds me i need to call the century link rep again [21:53:46] it's been almost a day since we talked [21:53:56] What have you done for me lately... [22:01:36] I'll be there in 2 minutes! [22:06:01] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, and 2 others: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2852488 (cwdent) [22:06:33] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Paypal, and 2 others: PayPal listener putting name in address fields - https://phabricator.wikimedia.org/T152510#2851138 (cwdent) a:cwdent [22:08:48] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: Central Notice: possible CN issue early on December 1st UTC - https://phabricator.wikimedia.org/T152122#2852493 (DStrine) [22:09:00] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325691 [22:09:11] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325691 (owner: Ejegg) [22:09:20] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/325691 (owner: Ejegg) [22:09:52] we need to file a bug for wrong cover sheets on the cps reports [22:15:44] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Figure out why new recurring entries are creating duplicate emails of the same location type, ditto address - https://phabricator.wikimedia.org/T152475#2849607 (awight) @Eileenmcnaughton Hum--It's possible that we're calling `wmf_civicrm_contribution_m... [22:15:52] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Figure out why new recurring entries are creating duplicate emails of the same location type, ditto address - https://phabricator.wikimedia.org/T152475#2852525 (awight) [22:16:08] log updated civicrm from d20c1c49ff99bdeb47dc183482ecc29abbc86280 to 610364c49c4598b2121346cdae73f8138c15bafa [22:16:50] Fundraising Sprint Waiting for Godot, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Figure out why new recurring entries are creating duplicate emails of the same location type, ditto address - https://phabricator.wikimedia.org/T152475#2852529 (ggellerman) p:Triage>... [22:18:16] Fundraising-Backlog, FR-Email: Spike: Investigate Silverpop invalid rows - https://phabricator.wikimedia.org/T152448#2852553 (ggellerman) p:Triage>Normal [22:19:30] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice choiceData should be filtered by country server-side - https://phabricator.wikimedia.org/T152251#2843138 (awight) Maybe--but then we need to split cache on country as well. [22:20:22] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice choiceData should be filtered by country server-side - https://phabricator.wikimedia.org/T152251#2852557 (ggellerman) p:Triage>Normal [22:52:26] fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#2852630 (fgiunchedi) [23:13:03] awight: when i try to cherry pick around 78a59db62eccfbc6cfa9ab08443e4cba31517ca4 i get some bizarre conflicts [23:13:25] d'argh [23:14:15] seems like it should be straightforward but i get conflicts from commits 6 months ago. wondering if it's stuff that was never on master. [23:14:52] (PS1) Awight: Clean up a couple unused things [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325703 [23:14:54] It worked fine for me [23:14:54] (PS1) Awight: Less noisy queue consumer logging [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325704 (https://phabricator.wikimedia.org/T151954) [23:14:57] (PS1) Awight: reject some address_names [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325705 (https://phabricator.wikimedia.org/T152510) [23:15:23] awight: ah did you use the UI? i was trying CLI [23:15:31] dunno what the difference would be [23:15:59] git checkout origin/deployment; git cherry-pick 5db2b2109071c9d642aefb09d7d15375f918551f e181e464087da17759f34b845fe7bd3995662e0b bbba45f67287ea94afd4eec5a982685ff33e23be [23:16:09] wtf [23:16:16] exactly what i did [23:16:17] It's your internet :p [23:16:19] i tried 3 times [23:16:54] Your patch gave me a conflict but only because of tests [23:18:28] yeah i got that one [23:18:49] but then it got angry about 872d29df2a70e2a4ab26e58a72cc7976de819648 [23:18:54] no idea why [23:20:02] hmm, yeah that's an old one [23:20:18] and then 05e4d46b2091550d122ad39ddfb52cb132e02cc7 [23:21:28] awight: you ok having no merge commit for these? [23:23:31] fine by me [23:23:48] How are you getting commit shas for the conflicts? [23:24:02] it sounds almost like you're merging with distant ancestors [23:24:28] My reflog looked like this: [23:24:28] f143378 HEAD@{0}: commit (cherry-pick): reject some address_names [23:24:28] 27c0b18 HEAD@{1}: cherry-pick: Less noisy queue consumer logging [23:24:28] 7c9ffaf HEAD@{2}: cherry-pick: Clean up a couple unused things [23:24:28] 36be698 HEAD@{3}: checkout: moving from deployment to origin/deployment [23:26:35] 54b7b93 HEAD@{7}: rebase: Kill tests on deployment [23:26:37] a88ff7b HEAD@{8}: cherry-pick: Less noisy queue consumer logging [23:26:39] 1caf208 HEAD@{9}: cherry-pick: Clean up a couple unused thing [23:27:01] i don't know why that rebase, i used cherry-pick [23:28:43] (CR) Cdentinger: [C: 2] Clean up a couple unused things [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325703 (owner: Awight) [23:28:48] (CR) Cdentinger: [C: 2] Less noisy queue consumer logging [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325704 (https://phabricator.wikimedia.org/T151954) (owner: Awight) [23:29:09] (CR) Cdentinger: [C: 2] reject some address_names [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325705 (https://phabricator.wikimedia.org/T152510) (owner: Awight) [23:29:58] (Merged) jenkins-bot: Clean up a couple unused things [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325703 (owner: Awight) [23:30:04] (Merged) jenkins-bot: Less noisy queue consumer logging [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325704 (https://phabricator.wikimedia.org/T151954) (owner: Awight) [23:30:21] (Merged) jenkins-bot: reject some address_names [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/325705 (https://phabricator.wikimedia.org/T152510) (owner: Awight) [23:30:40] cwd: what comes before that? [23:31:13] 36be698 HEAD@{10}: pull: Fast-forward [23:31:15] I'm on 2.11.0 fwiw [23:31:24] k that's the right commit [23:31:31] 2.8.1 [23:31:36] dafuq [23:32:13] how did you make "kill tests on deployment"--was that a local commit? [23:32:27] it's odd cos there shouldn't have been any tests in the repo at that point [23:33:53] this is getting even weirder [23:34:01] awight: git show 54b7b93d23009ac2953928335d4c49b6d6f618f5 [23:35:48] awight: https://sadbastard.org/what.png [23:37:05] ♩♫♩ do you see what i see? ♩♫♩ [23:38:19] i guess it'd be more like ♫♫♩♩ [23:38:56] O_o [23:39:04] I did not do [23:39:17] I don't have that commit [23:39:23] it's your commit message with the changes from my last one [23:39:37] yep [23:39:39] did something crash and burn in my git object store? [23:39:56] wowza [23:41:43] i have a wild hair to get to the bottom of this [23:41:51] yeah I'm glad that happened [23:42:02] but uh [23:42:07] maybe just export your commits and reclone [23:42:33] ... so you have a working copy and can do forensics offline [23:47:38] so weird! [23:47:42] upstream doesn't even have that commit [23:47:45] it's a chimera [23:51:18] awight: since this code change is executed by the jenkins job...which smashpig is that? the same as the listener? or do we need to update a submodule?