[00:15:35] fr-tech hmm I don't see standups in the calendar for the rest of the week, or for the last three days of any subsequent weeks... [00:31:46] AndyRussG: fixed [00:59:01] dstrine-away: ah K thx :) [00:59:19] (sorry for the bother, wasn't sure if there was some change I didn't know about...) [04:20:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:25:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:25:10] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:30:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:30:10] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:35:09] PROBLEM - check_disk on samarium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:35:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:35:19] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:40:09] PROBLEM - check_disk on samarium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:40:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:40:10] PROBLEM - check_disk on saiph is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:40:11] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:40:49] Jeff_Green: oh hey [04:45:09] PROBLEM - check_disk on samarium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:45:09] PROBLEM - check_disk on payments1004 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:45:10] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:45:10] PROBLEM - check_disk on saiph is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:45:11] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:50:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:50:10] PROBLEM - check_disk on samarium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:50:10] PROBLEM - check_disk on payments1004 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:50:11] PROBLEM - check_disk on saiph is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:50:11] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:50:16] i'm not sure why icinga is concerned with that dir [04:53:03] looks the same on complaining machines as non-complaining [04:55:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:55:10] PROBLEM - check_disk on samarium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:55:10] PROBLEM - check_disk on payments1004 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:55:11] PROBLEM - check_disk on saiph is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:55:11] PROBLEM - check_disk on payments2003 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:55:12] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [04:55:44] hey cwd [04:56:23] wassup [04:56:28] I suspect it's related to some of the recent kernel tweaks [04:56:51] seems likely [04:57:06] not positive [04:57:07] i guess i don't understand the icinga check [04:57:42] that dir looks to have the same perms as machines that haven't been updated [04:59:21] it's the same owner, but not the same perms on the two examples I looked at [04:59:35] here's a hack around it: http://www.dailyithelp.com/nagios-disk-critical-syskerneldebugtracing-is-not-accessible-permission/ [04:59:50] I just deployed that ^^^ everywhere to shut icinga up for now [05:00:09] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:09] PROBLEM - check_disk on samarium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:10] PROBLEM - check_disk on thulium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:10] PROBLEM - check_disk on betelgeuse is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:11] PROBLEM - check_disk on saiph is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:12] PROBLEM - check_disk on payments1004 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:12] PROBLEM - check_disk on payments2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:13] PROBLEM - check_disk on payments2003 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:00:49] RECOVERY - check_disk on payments2001 is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 12861 MB (99% inode=99%): / 49691 MB (93% inode=98%): /dev/shm 32196 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 32196 MB (100% inode=99%): /boot 192 MB (77% inode=99%) [05:02:18] looks like that's implemented in several spots in production puppet, so I'm guessing they hit it for the same reason [05:02:42] Jeff_Green: oh huh [05:03:16] you are right about the dir perms [05:04:32] I wonder which setting locked that down, there are a couple that seem likely [05:05:09] PROBLEM - check_disk on samarium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:05:11] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:05:11] PROBLEM - check_disk on betelgeuse is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:05:11] PROBLEM - check_disk on thulium is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:05:11] PROBLEM - check_disk on saiph is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:05:11] RECOVERY - check_disk on payments1004 is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 3166 MB (99% inode=99%): / 49565 MB (93% inode=98%): /dev/shm 7977 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7977 MB (100% inode=99%): /boot 190 MB (76% inode=99%): /srv 829859 MB (99% inode=99%) [05:05:12] RECOVERY - check_disk on payments2003 is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 12861 MB (99% inode=99%): / 49144 MB (92% inode=98%): /dev/shm 32196 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 32196 MB (100% inode=99%): /boot 192 MB (77% inode=99%) [05:07:01] Jeff_Green: you think from the module blacklist? [05:07:22] i was thinking the sysctl stuff [05:07:39] kernel.yama.ptrace_scope perhaps [05:08:46] that seems the most plausible to me [05:09:58] wasn't that a few days ago? [05:10:09] RECOVERY - check_disk on thulium is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 3153 MB (98% inode=99%): / 50272 MB (94% inode=98%): /dev/shm 7985 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 7985 MB (100% inode=99%): /boot 193 MB (77% inode=99%) [05:10:09] RECOVERY - check_disk on betelgeuse is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 1449 MB (90% inode=99%): / 50836 MB (95% inode=98%): /dev/shm 3986 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 3986 MB (100% inode=99%): /boot 191 MB (76% inode=99%): /srv 384416 MB (99% inode=99%) [05:10:10] RECOVERY - check_disk on samarium is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 6402 MB (99% inode=99%): / 51056 MB (95% inode=98%): /dev/shm 16049 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 16049 MB (100% inode=99%): /boot 193 MB (77% inode=99%) [05:10:11] PROBLEM - check_disk on pay-lvs2001 is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:10:11] PROBLEM - check_disk on saiph is CRITICAL: DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied [05:11:31] i rebooted all these today, maybe that had to happen for the permissions to change [05:12:17] oh that makes sense [05:12:31] although even that, hmm [05:12:47] why wouldn't they have started complaining right after reboot [05:14:07] maybe something scheduled that touches the dir? [05:15:09] RECOVERY - check_disk on pay-lvs2001 is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 6410 MB (99% inode=99%): / 25371 MB (47% inode=96%): /dev/shm 16068 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 16068 MB (100% inode=99%): /boot 194 MB (78% inode=99%) [05:15:09] RECOVERY - check_disk on saiph is OK: DISK OK - free space: /dev 10 MB (100% inode=99%): /run 6410 MB (99% inode=99%): / 50633 MB (94% inode=98%): /dev/shm 16068 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 16068 MB (100% inode=99%): /boot 194 MB (78% inode=99%) [05:17:47] i dunno, it's mysterious [05:19:35] ya [05:19:36] OH [05:19:43] duh. it's the new kernel I bet [05:19:52] https://bugs.launchpad.net/ubuntu/+source/nagios-plugins/+bug/1516451 [05:21:07] double duh, i guess same thing--if it were the kernel why did it take hours after boot before it blocked the nagios check [05:21:43] this def looks like the same thing tho [05:21:57] the delay is mysterious [05:22:23] yeah. I think I'm too tired to make real sense of this. The check_disk tweak should cause everything to recover [05:23:44] i'm gonna go back to bed, sorry for the pagerstorm and have a good night! [05:24:07] not at all, and you too! [11:43:39] morning all [11:45:31] hello! [12:04:45] hey ejegg :) [12:29:13] ejegg, when updating submodule pointers do you self merge? [12:30:51] jgleeson: yeah, usually [12:31:28] unless the libraries have big changes that need someone else's input [12:32:52] (PS1) Jgleeson: Update drupal submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/408791 [12:32:57] thanks [12:33:10] ah yeah, that one should be fine to self-merge [12:33:13] I'm pulling in the drupal fix for php7 [12:33:15] cool [12:33:43] (CR) Jgleeson: [V: 2 C: 2] "self merge!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/408791 (owner: Jgleeson) [12:34:43] I'm planning to deploy this commit to production in a SmashPig/vendor update: https://github.com/ejegg/login-and-pay-with-amazon-sdk-php/commit/e3f8d1082556104526999740beaf55b481aaf323 [12:34:52] LMK if it looks bad [12:35:38] eh, one whitespace fail, but that lib has atrocious whitespace inconsistencies already [12:38:23] I'll take a look! [12:38:32] thanks! [12:39:19] (Merged) jenkins-bot: Update drupal submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/408791 (owner: Jgleeson) [12:39:21] (PS1) Ejegg: Update Amazon SDK for selective TCP proxy [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/408794 (https://phabricator.wikimedia.org/T180168) [12:40:32] ah crap, Lenovo's recalling our laptop model [12:40:55] ok, guess not all of em, looking up my serial # [12:43:01] well crap [12:47:37] uh oh [12:47:57] where can I check/ [12:48:02] ? [12:48:19] https://support.lenovo.com/co/es/solutions/ht504453 [12:48:56] the amazon sdk update looks good to me [12:49:01] thanks! [12:49:12] (CR) Ejegg: [C: 2] Update Amazon SDK for selective TCP proxy [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/408794 (https://phabricator.wikimedia.org/T180168) (owner: Ejegg) [12:50:38] (Merged) jenkins-bot: Update Amazon SDK for selective TCP proxy [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/408794 (https://phabricator.wikimedia.org/T180168) (owner: Ejegg) [12:51:04] looks like I am one of the lucky ones [12:51:10] nice [12:51:28] (PS1) Ejegg: Update Amazon SDK for selective TCP proxy [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/408798 [12:51:35] that's hella inconvenient for others though [12:52:25] "Customers should immediately stop using affected computers" :S [12:53:00] (CR) Ejegg: [V: 2 C: 2] Update Amazon SDK for selective TCP proxy [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/408798 (owner: Ejegg) [12:55:01] you're talking at least a week, maybe 2 of downtime [12:55:21] hopefully the reship won't take 2 months like in my case [12:57:38] brb [13:34:32] ah man [13:34:52] getting the tests to work on stretch vagrant for donation interface is not a straightforward task [13:36:16] https://imgur.com/a/Yf4pB [14:04:28] jgleeson_: what's the biggest issue? [14:05:20] not entirely sure just trying to work through it now [14:05:28] ohhh, is that possibly just an issue with an outdated library? [14:05:54] like maybe a version of Smashpig without the FinalStatus class ? [14:06:36] so..I'm now comparing the db schema between wiki and payments wiki as there's lots of Error: 1146 Table 'drupal.unittest_contribution_tracking' doesn't exist (127.0.0.1) [14:06:41] in the errors [14:06:44] ahh [14:06:48] right, I got that too [14:07:10] it looks like the mw unit tests started using prefixed tables [14:07:13] so far I've run, git submodule foreach composer install [14:07:25] which installed lots of stuff [14:07:35] OK, so it probably is just that missing table [14:07:38] I'm wondering if there are related db updates missing [14:08:14] I guess we need to script out creating that unittest table too [14:08:31] how/where does that take place at current [14:08:45] we never needed it before [14:09:03] I just did it by hand to get past those errors this time [14:09:16] :S [14:09:28] yeah, annoying as heck [14:09:39] lemme see where the authoritative source for that table is [14:09:48] I think it might be in the drupal module [14:10:54] yeah, shoot, and it's in drupal-ese [14:11:04] i.e. not sql, but an array describing the schema [14:12:19] so... maybe 'mysql -e "show create table contribution_tracking" drupal | sed -e "s/contribution_tracking/unittest_contribution_tracking/" | mysql drupal' [14:13:32] ok will give that a go [14:14:00] also just ran `mwscript update.php --wiki=paymentswiki` which done a bunch of stuff [14:14:08] oh huh [14:14:24] i thought that all happened during the vagrant provision [14:14:37] sure it wasn't just checking all the things? [14:14:56] actually yeah after inspecting the output it looks like it's saying most of it is marked as already completed [14:19:02] ah, that command doesn't quite work, need to suppress the table formatting for the first bit [14:19:24] yeah and make it greedy [14:19:28] tweaking it as we speak [14:19:34] good starting point though [14:20:33] whoa, -H produces html table output! [14:20:50] lol [14:21:16] such an invitation to sql injection [14:21:28] shit, ejegg jgleeson, i got AFFECTED [14:21:35] mepps me too :( [14:21:51] at least shipping should be pretty quick for you [14:22:00] ejegg, so what's next step? i guess i need to stop using this [14:22:35] ugh, yeah. I guess contact Lenovo and see how fast they can fix it and turn it around [14:23:12] globally* [14:23:44] oh no! [14:30:47] yeah now i switched to my old work computer which is a mac and it's a weird adjustment [14:35:47] ejegg, does this look about right? [14:35:50] sudo mysql -e "show create table contribution_tracking" drupal | sed -ne 's/contribution_tracking/unittest_contribution_tracking/g' -Ee 's/^.*(CREATE.*)$/\1/p' [14:36:02] using sudo for mariadb root [14:36:08] testing* [14:39:25] hi mepps btw :) [14:39:34] hi jgleeson! [14:39:36] currently fighting with unit tests for payments wiki on vagrant [14:39:47] aww [14:39:53] I.WILL.MAKE.IT.WORK [14:39:57] :S [14:40:44] okay ejegg gone from 207 falures/errors to 3!!! [14:41:22] 33 skipped, due to: Orphan adapter not yet implemented [14:41:36] yep, and recurring not yet implemented [14:41:48] we could comment those out to make the output cleaner [14:42:04] then there are 2 due to not anticipating the protocol-relative URLs [14:42:14] i.e. //blah.org [14:42:59] yup just grepping the full list of ".*\s not implemented" [14:50:01] ok so the relative urls pair I see [14:50:34] the other is an adyen goodsSubmit test failure [14:50:36] - 'merchantReference' => '1.1' [14:50:37] + 'merchantReference' => '2.1' [14:50:52] oh weird [14:51:15] so there's an unexpected contribution_tracking number change [14:51:36] I wonder if that has anything to do with the unittest_ table [14:54:41] are these 3 tests worth fixing in vagrant before I push a patch to add the unittest_contribution_tracking table ? [14:55:47] Seems worth pushing that patch first [14:58:11] it's not a consistent failure, that particular test [14:58:13] - 'merchantReference' => '1.1' [14:58:13] + 'merchantReference' => '734.1' [14:58:38] hmm [14:58:48] weird, I wonder if we're using one table for the expected and the other table for the acutal [14:58:56] so in relation to the missing table, we only want this when unit tests are run right? [14:59:18] so it doesn't make sense to put it in with the fundraising role [14:59:25] as we don't bring down the tests [14:59:26] hmm [15:00:09] it feels like a bootstrap function of the drupal tests [15:00:16] on master [15:00:35] if it's drupal specific [15:01:11] well, it's really DonationInterface specific [15:01:34] it's only hitting drupal because of the dumb dependency that we hope to kill off this year [15:02:10] and since vagrant is really just a dev tool for us, I'd be fine with creating the unittest table up front if that's the easiest way to do it [15:02:55] on the other hand, if the rest of the unittest_ tables (in the wiki db) are being created and destroyed at the start/end of test runs, we could look into how that happens [15:03:11] and why it's getting ours wrong [15:03:46] Yeah. I guess if we want to do it all proper, it means digging into the mediawiki test harness [15:04:14] and if we want to take the command-line/sed shortcut, we might as well just set it up at the provision stage [15:04:38] especially for this dependency we want to kill within the year [15:06:30] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/408815 [15:06:35] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/408815 (owner: Ejegg) [15:07:00] \DonationInterfaceTestCase::verifyFormOutput [15:07:05] within this [15:07:16] hehe, that one's interesting [15:07:24] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/408815 (owner: Ejegg) [15:07:27] almost browser testing! [15:08:32] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, and 5 others: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3952759 (DStrine) [15:08:49] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Adding translatable message to banner FR2015_translations takes a long time and shows error - https://phabricator.wikimedia.org/T167442#3952761 (DStrine) [15:09:33] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: "Internal error" on testwiki's Special:LanguageStats due to non-existing CentralNotice banner - https://phabricator.wikimedia.org/T157997#3952765 (DStrine) [15:10:05] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, Wikimedia-Site-requests: Change (centralnotice-preview-all-template-translations) link target - https://phabricator.wikimedia.org/T158508#3952767 (DStrine) [15:10:18] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, I18n: Can't use Translate extension's variables feature in CentralNotice - https://phabricator.wikimedia.org/T87448#3952768 (DStrine) [15:10:36] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, Easy, I18n: CentralNotice reloads old translatable messages on form submit - https://phabricator.wikimedia.org/T72939#3952769 (DStrine) [15:10:56] woah [15:11:25] mediawiki-fr/tests/phpunit/MediaWikiTestCase.php [15:11:58] * ejegg follows the white rabbit [15:12:47] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, Epic, I18n: [Epic] CentralNotice translation should move closer to MediaWiki i18n standards and the code cleaned up - https://phabricator.wikimedia.org/T116235#3952774 (DStrine) [15:12:57] that is one BUSY test base [15:13:09] I could loose hours xdebugging that [15:13:11] heh, yeah [15:13:19] I'll go with scripted creation [15:13:24] got that telltale DB_PREFIX='unittest_' [15:13:30] :) [15:13:31] Fundraising Sprint N*E*R*D, Fundraising Sprint ODB, Fundraising Sprint Snoop (Dogg|Lion), Fundraising-Backlog, and 4 others: Publishing translations for central notice banners fails - https://phabricator.wikimedia.org/T104774#3952775 (DStrine) [15:13:37] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: BUG: CentralNotice banner translations sometimes go missing from Translate interface - https://phabricator.wikimedia.org/T90863#3952776 (DStrine) [15:13:49] Fundraising Sprint Snoop (Dogg|Lion), Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Central Notice message groups are slow to index - https://phabricator.wikimedia.org/T111189#3952777 (DStrine) [15:13:52] ye I saw that and though, ok lets see how that works... and then saw the rest [15:13:57] thought* [15:14:03] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, I18n: Allow CentralNotice banner variables to be marked as non translated - https://phabricator.wikimedia.org/T53470#3952778 (DStrine) [15:14:19] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, Easy: CentralNotice: "Preview all approved translations" is dead - https://phabricator.wikimedia.org/T105558#3952779 (DStrine) [15:16:18] mepps I'm going to deploy DonationInterface with that ingenico update [15:25:59] great ejegg [15:26:34] just trying to test the latest proxy thing locally, but VPN isn't letting me on for some reason [15:26:44] oh boo ejegg [15:31:40] oh huh, commandline worked [15:31:52] nmcli con up id [15:34:55] cool, new config seems to be happy [15:35:11] (PS1) Ejegg: Update Amazon SDK for selective TCP proxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408820 [15:35:39] (CR) Ejegg: [C: 2] Update Amazon SDK for selective TCP proxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408820 (owner: Ejegg) [15:37:25] (Merged) jenkins-bot: Update Amazon SDK for selective TCP proxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/408820 (owner: Ejegg) [15:38:34] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408821 [15:39:28] (CR) jerkins-bot: [V: -1] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408821 (owner: Ejegg) [15:40:05] (PS1) Ejegg: Update Amazon SDK for selective TCP proxy [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/408822 [15:40:20] (CR) Ejegg: [V: 2 C: 2] Update Amazon SDK for selective TCP proxy [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/408822 (owner: Ejegg) [15:41:02] (PS2) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408821 [15:50:25] ejegg, so the other unittest_ prefix tables are generated by a sideways copy of the wiki database however as contribution_tracking is part of the drupal db it gets ignored [15:51:44] mediawiki-fr/tests/phpunit/MediaWikiTestCase.php:703 [15:51:55] jgleeson_: ah, thanks [15:52:30] yeah, that ContributionTracking extension is weird with its own db connection and should go away [15:53:02] morning! :) [15:53:04] just wondering how best to add it in as the obvious idea of just adding the script to generate and create the database to the puppet build process doesn't work, as puppet can't deal with return values from the sed command [15:53:13] hey AndyRussG ;) [15:53:22] jgleeson_: :) [15:53:38] hi AndyRussG ! [15:53:38] puppet blocks don't return any values for that matter [15:53:42] ejegg: :) [15:54:09] jgleeson_: can you just pipe the return value of sed into another sudo mysql drupal ? [15:54:22] on the same command line [15:56:23] ejegg are you seeing all this smashpig amazon failmail? [15:56:53] yeah [15:57:06] shoot, I thought the settings rollback I did last night would fix it [15:57:21] but since it didn't, I'm going to deploy the next code fix [15:58:20] Yeah, that works although the payments wiki module in puppet handles the wiki and extensions including donation interface. I was originally just going to drop it in donations interface puppet/modules/payments/manifests/donation_interface.pp but as it is a table relating specifically unit tests on another db (drupal), it feels like the wrong place for and I'm not sure how it would stand up to code review. [15:58:43] jgleeson_: I'd just put it in the crm module [15:58:52] ah ok [15:58:59] that might be a better choice [15:58:59] since it has to depend on the crm install being completed [16:00:14] sure, it feels a bit wrong either place, but hopefully we can get rid of it in a few months [16:00:58] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408821 (owner: Ejegg) [16:03:05] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/408821 (owner: Ejegg) [16:06:04] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/408825 [16:06:14] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/408825 (owner: Ejegg) [16:11:17] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/408825 (owner: Ejegg) [16:17:40] fr-tech I'm gonna be afk for a little while, I need to buy a stamp and post a letter before 5 so will be back shortly [16:17:45] k [16:17:51] damn snail mail ! [16:34:55] fr-tech i'm pretty blocked by this laptop issue today, i thought i had an environment set up on one of my other two laptops (one is a very old personal machine, one my old work computer) but so far i'm not finding it setup on either machine [16:36:32] i'm waiting for a call back from the local service folks, just hoping this thing doesn't catch fire :( [16:40:03] i'm on hold with them now [16:43:25] oh man BUMMER [16:43:30] mine is affected as well [16:44:42] * cwd backs up home dir [16:49:03] i am really hating this hold--they keep interrupting the music every minute to tell me i'm still on hold [16:49:18] mepps: cwd: what happened with the laptops? [16:49:28] they were recalled AndyRussG [16:49:30] There was a recall issued [16:49:32] they might overheat! [16:49:37] oh drat [16:49:41] battery might splode! [16:49:51] https://support.lenovo.com/co/es/solutions/ht504453 [16:49:54] so what I can't use it? [16:49:58] can check your serial # ^^^ [16:50:42] Hmmm [16:50:48] Maybe I won't... [16:50:52] I didn't see anything [16:52:24] Hmmm my "machine type" isn't in the drop-down list [16:53:10] Mine says "20FB" [16:53:20] * AndyRussG sighs with relief [16:54:31] the funniest part for me is my truck had a recall last year for the lethal takata inflators, and they were both replaced, but now there is a recall on that recall, but the news parts are not available yet [16:54:47] hilarious... [16:54:50] so i'm just supposed to not drive it [16:55:01] which might make taking the laptop in for recall a challenge [16:55:04] * cwd slaps knee [16:55:50] heheheh [16:56:19] Well I guess you gotta evaluate the relative risks and significance of the consequence [16:56:30] I think there are philosophy textbooks about this [16:56:59] Also: is there anything improper about having more than one "really" in a method name? [16:57:37] As in, recordImpression() calls reallyRecordImpression() which calls reallyReallyRecordImpression() [16:57:40] Just askin' [16:59:05] Wonder if there's a correlation between the number of jokes in code and its maintainability (or lack thereof) [16:59:24] it appears that there are no service providers anywhere near here [16:59:32] can i just screw this screw in? [16:59:42] right? [16:59:52] Do you know which screw it is? [17:00:07] aren't thinkpad's reknowned for field-replacable units [17:00:07] There might be a few [17:00:37] probably not since they ditched the ethernet jack in favor of the wafer thin form factor [17:23:29] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Ingenico, FR-Smashpig, Fr-Ingenico-integration_2017-18: Donation queue consumer: support alternate ID for pending row - https://phabricator.wikimedia.org/T163951#3953303 (Ejegg) [17:23:41] Wikimedia-Fundraising: Create translated versions of new Thank You page - https://phabricator.wikimedia.org/T185333#3953306 (schoenbaechler) As discussed in our campaigns call yesterday, I transferred all translation strings in English into a Google Spreadsheet: https://docs.google.com/spreadsheets/d/1rM0kl... [17:23:44] ok sounds like it's actually a loose screw that just needs to be removed [17:25:33] it can damage the battery so it must be near the battery [18:01:22] PROBLEM - Host heka is DOWN: PING CRITICAL - Packet loss = 100% [18:02:02] ^ this and the puppet-related pagerstorm are expected, I'm rebooting heka for some kernel tweaks [18:05:22] RECOVERY - Host heka is UP: PING OK - Packet loss = 0%, RTA = 36.22 ms [18:19:49] sigh fr-tech, so much of today is being spent on hold [18:33:43] fr-tech any news for scrum of scrums? [18:34:03] ejegg not here, thanks much! [18:40:09] PROBLEM - check_puppetrun on heka is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[isc-dhcp-server] [18:45:09] RECOVERY - check_puppetrun on heka is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:10:13] PROBLEM - check_puppetrun on heka is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[isc-dhcp-server] [19:15:15] RECOVERY - check_puppetrun on heka is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [19:43:15] !log updated payments-wiki from 39a7ef32e5 to fe311c2d26 [19:43:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:47:21] !log updated SmashPig from 1f56978c0c to 1ebee97a45 [19:47:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:04:24] ejegg: can I get you to merge another instance of that globals test snaffu - https://gerrit.wikimedia.org/r/#/c/408715/ [20:08:29] (PS2) Ejegg: Fix another place where global resetting can lead to e-notice [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/408715 (owner: Eileen) [20:08:35] (CR) Ejegg: [C: 2] Fix another place where global resetting can lead to e-notice [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/408715 (owner: Eileen) [20:12:43] thanks ejegg [20:12:49] yw! [20:13:13] (CR) Eileen: "recheck" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/408690 (owner: Eileen) [20:18:13] (PS3) Ejegg: Use ValidationAction constants from SmashPig [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407564 (https://phabricator.wikimedia.org/T163868) [20:19:06] turning over my work laptop for the next day or two fr-tech :( [20:19:45] (CR) jerkins-bot: [V: -1] CRM-21738 fix transfer of view_only custom data. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/408690 (owner: Eileen) [20:19:54] ugh [20:20:07] still waiting on the call back here [20:20:19] hmm [20:26:56] (CR) Ejegg: [C: 2] "Kicking zuul" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407564 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [20:27:26] (PS4) Ejegg: Use PaymentError from SmashPig [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407565 (https://phabricator.wikimedia.org/T163868) [20:27:34] woot - down to one fail on the CiviCRM 4.7.31 stuff (although I don't think it fails on it's own - sigh) [20:27:34] (PS2) Ejegg: Use ValidationError from SmashPig [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407566 (https://phabricator.wikimedia.org/T163868) [20:29:32] (Merged) jenkins-bot: Use ValidationAction constants from SmashPig [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407564 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [20:33:18] how're the teef XenoRyet ? [20:42:16] (CR) Ejegg: [C: 2] "Restoring mepps's +2 to convince zuul to merge" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407565 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [20:42:23] lol [20:42:29] having a little battle there with Zuul [20:42:38] heh, what else is new? [20:42:47] I liked "Kicking Zuul" [20:43:51] Same as always. Just wish the only good dentist around here wasn't so far away. [20:43:59] Takes forever to get over there and back. [20:45:00] I need to visit the Dentist also. They sent me a reminder in December :S [20:46:14] (Merged) jenkins-bot: Use PaymentError from SmashPig [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407565 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [20:46:16] (Merged) jenkins-bot: Use ValidationError from SmashPig [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/407566 (https://phabricator.wikimedia.org/T163868) (owner: Ejegg) [20:49:16] Dentists are evil [20:49:25] (proposed sprint name) [20:50:31] heh [20:55:03] Mine is like a dental ninja. Last time I had a cavity filled, he said he could do it without novicane so I wouldn't be numb all day. Went in without any anesthetic at all and I swear I still didn't feel a thing. [20:55:35] He's like "I just won't touch any nerves, it'll be fine." [20:55:40] And he pulled it off. [21:01:43] !_! [21:02:24] cwd do you know anything about making rsyslog happy with python's SysLogHandler format? [21:02:33] https://bugs.python.org/issue12168 [21:03:16] no, it says fixed? [21:03:51] oh huh [21:04:08] ahh, I guess just in python 3.2+ [21:04:39] we're still using 2.7 for things like the paypal parser, right? [21:04:41] i have just been trying to get my old laptop working [21:05:34] i dug around in the X1 a little but no obvious screw [21:07:06] rgh, really want to use that patch to get rid of custom logger [21:09:24] default is 2.7 [21:09:30] but 3.4 is available [21:09:49] yeah, I think our code is not quite compatible [21:15:08] aah [21:30:45] fr-tech, I've pushed a patch (if you can call it that) for MediaWiki/vagrant to fix the issues preventing us unit testing paymentswiki and donation interface. If anyone running vagrant has some time and would like to test it out and confirm you can you can run the tests out of the box, the patch is here https://gerrit.wikimedia.org/r/#/c/408926/ [21:31:09] -you can [21:31:19] jgleeson: I'm on it [21:31:19] thanks! [21:34:26] on that note I'm signing off, catch you all tomorrow o/ [21:35:06] PROBLEM - check_puppetrun on pay-lvs1002 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 7 minutes ago with 32 failures. Failed resources (up to 3 shown) [21:35:06] PROBLEM - check_puppetrun on frauth1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 5 minutes ago with 26 failures. Failed resources (up to 3 shown): File[/etc/motd.tail],File[/usr/bin/check-raid.py],File[/etc/rssh.conf],File[/usr/local/bin/yubikey_otp_filter] [21:35:06] PROBLEM - check_puppetrun on tellurium is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 7 minutes ago with 27 failures. Failed resources (up to 3 shown) [21:35:06] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 5 minutes ago with 30 failures. Failed resources (up to 3 shown) [21:35:06] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 7 minutes ago with 40 failures. Failed resources (up to 3 shown) [21:35:13] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 7 minutes ago with 29 failures. Failed resources (up to 3 shown) [21:35:13] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 7 minutes ago with 29 failures. Failed resources (up to 3 shown) [21:40:04] RECOVERY - check_puppetrun on frauth1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [21:40:04] RECOVERY - check_puppetrun on pay-lvs1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:40:05] RECOVERY - check_puppetrun on tellurium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:40:13] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 10 minutes ago with 30 failures. Failed resources (up to 3 shown) [21:40:13] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:40:14] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 12 minutes ago with 40 failures. Failed resources (up to 3 shown) [21:40:14] PROBLEM - check_puppetrun on frdb1001 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 12 minutes ago with 29 failures. Failed resources (up to 3 shown) [21:45:13] RECOVERY - check_puppetrun on frlog1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:45:13] RECOVERY - check_puppetrun on payments1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:45:15] RECOVERY - check_puppetrun on frdb1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [21:45:52] fr-tech: i have some family coming over so i'm out for a bit, back this evening [21:53:18] (CR) XenoRyet: [C: 2] "Code looks good to me. Mapping choices look right as far as I'm aware. I agree with leaving special instructions unmapped for now, we ca" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/401819 (https://phabricator.wikimedia.org/T179339) (owner: Eileen) [21:58:58] (Merged) jenkins-bot: Citibank import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/401819 (https://phabricator.wikimedia.org/T179339) (owner: Eileen) [22:02:50] thanks @xenoRyet [22:04:47] No worries [22:08:06] Fundraising Sprint Asymmetrical Earth Theory, Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Winter Wanderland, and 2 others: make a wiki page f... - https://phabricator.wikimedia.org/T181652#3797151 [23:23:11] good news - got all but one of my patches into 4.7.31 rc - means less reviewing for fr-tech :-) [23:23:24] Nice [23:23:57] sweet! [23:24:30] some were upstreaming but also a few pretty big geocoding patches [23:25:07] which you will be glad not to have to know about. I missed out on the one to get smart groups into proximity search - but all the preliminary ones on that went in [23:25:36] I'll get the new code onto staging today & see how long the upgrade takes [23:33:12] wow, cool! [23:37:21] ohh I might even get that last one snuck in - I convinced someone to review it