[00:23:23] PROBLEM - check_disk on bellatrix is CRITICAL: DISK CRITICAL - free space: /dev 80567 MB (100% inode=99%): /run 16043 MB (99% inode=99%): / 3878 MB (10% inode=98%): /dev/shm 80580 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 80580 MB (100% inode=99%): /srv 9850291 MB (64% inode=99%) [00:28:13] PROBLEM - check_disk on bellatrix is CRITICAL: DISK CRITICAL - free space: /dev 80567 MB (100% inode=99%): /run 16043 MB (99% inode=99%): / 3872 MB (10% inode=98%): /dev/shm 80580 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 80580 MB (100% inode=99%): /srv 9850291 MB (64% inode=99%): /run/user/12179 16116 MB (100% inode=99%) [00:28:18] blargh [00:31:26] i think the qualys scan is coming from a new IP [00:31:33] which is causing firewall logspam [00:33:23] PROBLEM - check_disk on bellatrix is CRITICAL: DISK CRITICAL - free space: /dev 80567 MB (100% inode=99%): /run 16043 MB (99% inode=99%): / 3871 MB (10% inode=98%): /dev/shm 80580 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 80580 MB (100% inode=99%): /srv 9850291 MB (64% inode=99%): /run/user/12179 16116 MB (100% inode=99%) [00:38:23] PROBLEM - check_disk on bellatrix is CRITICAL: DISK CRITICAL - free space: /dev 80567 MB (100% inode=99%): /run 16043 MB (99% inode=99%): / 3870 MB (10% inode=98%): /dev/shm 80580 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 80580 MB (100% inode=99%): /srv 9850291 MB (64% inode=99%): /run/user/12179 16116 MB (100% inode=99%) [00:43:24] PROBLEM - check_disk on bellatrix is CRITICAL: DISK CRITICAL - free space: /dev 80567 MB (100% inode=99%): /run 16043 MB (99% inode=99%): / 3869 MB (10% inode=98%): /dev/shm 80580 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 80580 MB (100% inode=99%): /srv 9850291 MB (64% inode=99%): /run/user/12179 16116 MB (100% inode=99%) [00:48:23] PROBLEM - check_disk on bellatrix is CRITICAL: DISK CRITICAL - free space: /dev 80567 MB (100% inode=99%): /run 16043 MB (99% inode=99%): / 3868 MB (10% inode=98%): /dev/shm 80580 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 80580 MB (100% inode=99%): /srv 10000563 MB (65% inode=99%): /run/user/12179 16116 MB (100% inode=99%) [00:51:03] fundraising-tech-ops, Operations, netops: Qualys scans causing problematic pfw logspam - https://phabricator.wikimedia.org/T206431 (cwdent) [00:53:23] RECOVERY - check_disk on bellatrix is OK: DISK OK - free space: /dev 80567 MB (100% inode=99%): /run 16043 MB (99% inode=99%): / 22395 MB (63% inode=98%): /dev/shm 80580 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 80580 MB (100% inode=99%): /srv 10000563 MB (65% inode=99%): /run/user/12179 16116 MB (100% inode=99%): /run/user/571 16116 MB (100% inode=99%) [01:03:25] cwd: hey :) I didn't do much, glad nothing broke (so far ;) ) [02:54:46] fundraising-tech-ops, Operations, Traffic, HTTPS: Re-evaluate use of EV certificates for payments.wm.o? - https://phabricator.wikimedia.org/T204931 (Liuxinyu970226) @krenair please, no more DV certs, that's the reason why jawiki, ugwiki, wuuwiki, zhwiki, zh-yuewiki and zhwikinews are SNI RSTed by... [14:48:58] hey ejegg, hows it going? I see we got a wave of failmail over the weekend. Looks like it's back to normal now? [15:00:55] hi jgleeson [15:01:19] yep, AndyRussG figured out it was triggered by a monthly disk check [15:01:48] that happened to run at the same time as the nightly WX audit parser [15:02:12] Which seems to have been looking in logs back to 2011 for donor data [15:02:52] I think that must be someone using an old reference number to do a bank transfer [15:03:05] can't think of any other reason there would be a payment that old in the audit [15:03:23] yeah that is pretty far back [15:03:49] anyway, with those two things fighting for disk access, anything else wanting to read and write files just hung for minutes and minutes [15:04:10] how's your day going? [15:05:47] actually I didn't figure it out, volans did, I just checked some stuff to confirm his theory [15:06:06] ah, well, you asked the right person, anyway! [15:07:42] ah it's goin okay. I feel like I could do with talking through some of the FRUEC work with someone more experienced with Python than myself as I'm stalling a little with my current approach. I'm about to go have dinner now though but if you or AndyRussG are around later it would be great to do a quick call and talk through where I'm stuck. [15:08:02] sure! [15:08:23] can't say I'm a the biggest python expert, but I'd be happy to chat [15:10:02] yeah, even just a fresh set of eyes to evaluate what I have so far. I'll push up a WIP version and we could talk through it. I'm using a couple of sets to try and compare data but the results aren't what I expect [15:10:23] when I'm back I'll ping you if thats ok [15:11:00] catch you later! [15:49:44] fr-tech it looks like we should pull that locale bug into sprint - getting plenty of failmail about it [15:51:04] Fundraising Sprint They Live, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, MediaWiki-extensions-DonationInterface: Invalid locale sent to new Ingenico API - https://phabricator.wikimedia.org/T206093 (Ejegg) Getting lots of failmail. This should be pretty straightforward, so let's pull it... [15:54:12] weird, looks like DonorLocale is already in the transformers list, and language is in var_map [15:54:21] so why isn't that coming through right? [16:00:44] (PS1) Ejegg: Fix duplicate key in YAML [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465188 [16:35:22] (PS1) Ejegg: Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) [16:37:24] (CR) jerkins-bot: [V: -1] Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [16:37:48] (PS2) Ejegg: Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) [16:38:28] Fundraising Sprint They Live, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, MediaWiki-extensions-DonationInterface, Patch-For-Review: Invalid locale sent to new Ingenico API - https://phabricator.wikimedia.org/T206093 (Ejegg) p:Triage>High a:Ejegg [16:39:45] (CR) jerkins-bot: [V: -1] Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [16:43:40] (PS3) Ejegg: Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) [16:44:06] fr-tech ok, that should be ready for review ^^^ [16:57:16] Fundraising Sprint They Live, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, MediaWiki-extensions-DonationInterface, Patch-For-Review: Invalid locale sent to new Ingenico API - https://phabricator.wikimedia.org/T206093 (Ejegg) Oh hey, we're getting some errors where the language code s... [17:13:36] boo, none of the php_intl functions seem to know how to take 'yue' and output a xx_YY string that makes sense [17:24:54] Wikimedia-Fundraising, Operations, SRE-Access-Requests: Frdev1001 server and mysql access - https://phabricator.wikimedia.org/T206478 (Krenair) pgehres hasn't worked here for years. anyway, adding this to #SRE-Access-Requests and shifting the fundraising project from subscribers to tags [17:26:26] Fundraising Sprint They Live, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, MediaWiki-extensions-DonationInterface, Patch-For-Review: Invalid locale sent to new Ingenico API - https://phabricator.wikimedia.org/T206093 (Ejegg) So, the [[ http://www.iana.org/assignments/language-subtag-... [18:56:56] hi jgleeson! [18:59:22] I'm available to chat now - or if 20 min from now is good for you, I'll just head to a cafe and be ready then [19:00:57] hey ejegg, just finishing a review on your patch, be done in 1 minute, or I'm around for a bit longer if you'd rather head the cafe [19:01:20] ok, cool, think I will head to the cafe then [19:02:27] (CR) Jgleeson: [C: 1] "Looks good, tests passing for me." (2 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [19:12:50] (PS1) Jgleeson: WIP: add in some useful comparison stats alongside backend output [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/465203 [19:18:47] (PS2) Jgleeson: WIP: add in some useful comparison stats alongside backend output Change-Id: I184e93d9a3eb9985b2b816d04a2cd6c7ddbfdd69 [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/458928 [19:23:54] (PS2) Jgleeson: WIP: add in some useful comparison stats alongside backend output Change-Id: Ibecd1d4c7de8e88ca97771d1aae1004cc9fc91da [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/465203 [19:24:36] (Abandoned) Jgleeson: WIP: add in some useful comparison stats alongside backend output Change-Id: I184e93d9a3eb9985b2b816d04a2cd6c7ddbfdd69 [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/458928 (owner: Jgleeson) [19:24:56] (Abandoned) Jgleeson: WIP adding in comparison summary using sql temp tables data [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/458929 (owner: Jgleeson) [19:25:26] hi again jgleeson [19:25:30] ok, more like 25 min [19:25:41] howdy! [19:25:56] queenmary good? [19:26:03] sure, I'll be right there [19:50:23] (Restored) Jgleeson: WIP adding in comparison summary using sql temp tables data [wikimedia/fundraising/FRUEC] - https://gerrit.wikimedia.org/r/458929 (owner: Jgleeson) [19:51:28] (PS4) Ejegg: Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) [20:35:24] (PS5) Ejegg: Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) [20:37:14] (CR) Ejegg: "Thanks jgleeson. I think this is the best we can do without pulling in another library or hardcoding specific examples as they come up. Le" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [20:44:01] (CR) Jgleeson: [C: 2] "got it, makes sense to me." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [20:45:08] have a good one fr-tech! [20:45:41] (Merged) jenkins-bot: Fix staging for donor locale [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/465192 (https://phabricator.wikimedia.org/T206093) (owner: Ejegg) [20:51:08] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/465209 [20:51:19] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/465209 (owner: Ejegg) [20:52:55] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/465209 (owner: Ejegg) [20:55:41] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/465211 [20:55:45] (CR) Ejegg: [C: 2] Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/465211 (owner: Ejegg) [20:59:33] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_27) - https://gerrit.wikimedia.org/r/465211 (owner: Ejegg) [21:11:17] !log updated payments-wiki from d623de9494 to 1472604b6e [21:11:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:14:47] Fundraising Sprint They Live, Fundraising-Backlog, Fr-Ingenico-integration_2017-18, MediaWiki-extensions-DonationInterface, and 2 others: Invalid locale sent to new Ingenico API - https://phabricator.wikimedia.org/T206093 (Ejegg) Did I really say 'pretty straightforward' on a human-language-relat... [21:53:21] (CR) jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/465231 (owner: L10n-bot) [21:55:42] Fundraising-Backlog, FR-Smashpig: Retry Ingenico API calls on (some) 5xx status codes? - https://phabricator.wikimedia.org/T206492 (Ejegg) [21:57:54] (PS1) Ejegg: No failmail on retry response. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/465285 [22:01:16] (PS1) Ejegg: Retry API requests on 502 Proxy Error [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/465286 (https://phabricator.wikimedia.org/T206492) [22:34:21] Fundraising Sprint They Live, Fundraising-Backlog, FR-Smashpig, Patch-For-Review: Retry Ingenico API calls on (some) 5xx status codes? - https://phabricator.wikimedia.org/T206492 (Ejegg) p:Triage>Low [22:34:53] Fundraising Sprint They Live, Fundraising-Backlog, FR-Smashpig, Patch-For-Review: Retry Ingenico API calls on (some) 5xx status codes? - https://phabricator.wikimedia.org/T206492 (Ejegg)