[13:55:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1255 [14:00:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1347 [14:05:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1412 [14:10:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1478 [14:15:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1673 [14:16:43] hmmm [14:16:50] looks dedupe related [14:18:30] does not normally cause replag [14:20:12] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1847 [14:20:23] ACKNOWLEDGEMENT - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1847 Jeff_Green known looking at it [14:21:09] Jeff_Green: weird that it's on 1002 but not 2001? [14:22:23] yeah, or it's just weird anyway, also frdb2001 is back to normal after the RAID repair last week [14:23:23] did you see that dedupe job that was cronspamming? [14:23:33] i saw the process yeah [14:24:09] i wonder if it's related? [14:25:07] i'm pretty sure this is 100% dedupe [14:25:15] but i don't get the asymmetrical replag [14:25:39] ooooh sh*T i know what's going on [14:26:12] ? [14:26:21] remember when we restarted the slaves and performance returned? we didnt' restart 1002 cause it wasn't lagging, it had been restarted more recently than the other two [14:26:53] yeah that's a possibility, although I think I had restarted it not long before that [14:27:05] yeah exactly [14:27:10] sorry just the other one i mean [14:27:15] 2001 was lagging like crazy [14:27:34] lets see what uptime is [14:27:41] bet you if we restart mysql on 1002 it will clear right out [14:28:18] we tried this though [14:28:25] ? [14:28:36] well that's true it did clear faster [14:28:45] let's gather as much info as we can before we go that route [14:30:08] comparing frdev and 1002, frdev has been up 7 days, 1002 up 9 days, and dev recovered fast [14:30:24] ah right frdev was the other one [14:30:31] still on my first cup of coffee [14:30:46] it would be amazing if a dba was around [14:30:50] 2001 has been up less than 24H [14:33:55] interesting, did you restart that? [14:33:58] i don't think i did [14:42:38] sorry, there was IRL interruption [14:42:51] we stopped it for the networking thing [15:05:13] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:10:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:15:13] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:20:13] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:24:37] fundraising-tech-ops, Operations, netops, Patch-For-Review: Move codfw frack to new infra - https://phabricator.wikimedia.org/T171970#3512641 (ayounsi) [15:25:13] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:30:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:35:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:40:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:45:13] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:50:12] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 790822 Threads: 2 Questions: 24162137 Slow queries: 4776 Opens: 14072 Flush tables: 1 Open tables: 602 Queries per second avg: 30.553 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [15:50:22] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:55:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [15:56:03] eileen: bizarre replag problem again today, but i don't think it has anything to do with the dedupe timeouts [15:56:11] also i don't even understand what time it is there [16:00:22] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [16:05:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [16:09:13] cwd: are you going to wikimania? [16:10:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [16:10:57] dstrine: nope :( [16:11:02] schedule conflicts [16:11:06] yeah me neither [16:11:42] cwd: I am prepping for the Ham radio technician license. did you say you were looking into it too? [16:12:11] awesome [16:12:16] not actively but i've thought about it for a long time [16:12:43] the intro level seems really easy [16:13:03] i didn't realize you don't need morse code anymore [16:13:13] heh [16:13:25] yeah i've heard you just memorize the questions morning of the test [16:14:06] yeah but then you're not really a good community member [16:14:14] and skills build on each other [16:14:42] so you are really just hurting yourself [16:15:12] PROBLEM - check_puppetrun on mintaka is CRITICAL: CRITICAL: Puppet last ran 1 day ago [16:16:25] well, i just mean a lot of the technical stuff you will never have to remember after the test [16:20:12] RECOVERY - check_puppetrun on mintaka is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:39:27] ejegg: u@wikimania? [17:03:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi imports currently aren't working. - https://phabricator.wikimedia.org/T172918#3513186 (LeanneS) [17:04:28] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi imports currently aren't working. - https://phabricator.wikimedia.org/T172918#3513198 (LeanneS) p:Triage>High [17:09:44] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi imports currently aren't working. - https://phabricator.wikimedia.org/T172918#3513222 (LeanneS) I also just tried another import type and it also didn't work, so it appears that there is no import functionality right now. [17:09:56] cwd I wonder if the dedupe just needs a bit of tuning - ie. not time out after 20 minutes. I think there is code so a new process won't start if one is going [17:10:49] so it might just be we need to let it have more time if it wants [17:19:10] fundraising-tech-ops, Operations, netops: bonded/redundant network connections for fundraising hosts - https://phabricator.wikimedia.org/T171962#3513283 (Jgreen) Let's do some initial testing using mintaka and frdb2001. It should be fine to connect the second interface on each of those anytime, since... [17:24:06] cwd yep, here in Montreal [17:24:23] looking forward to checking out the expo 77 grounds [17:25:35] 76 [17:25:55] Ah check out the botanical gardens, eh, right around there [17:26:45] Sorry, expo 67 I think, olympics were 76 [19:15:10] dstrine: am i able to connect a parent task and subtask? if not i noticed these two should probably be linked: https://phabricator.wikimedia.org/T170314 https://phabricator.wikimedia.org/T166613 [19:17:40] or ejegg you might know this ^^ [19:36:58] ah this is kind of an annoying UX [19:37:06] but it is possible [19:37:41] mepps: which is the subtask? [19:39:18] dstrine https://phabricator.wikimedia.org/T170314 [19:39:52] that one is the subtask [19:40:41] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Update DonationInterface code style for php5.5 - https://phabricator.wikimedia.org/T166613#3513892 (DStrine) [19:40:57] ok I just linked them. [19:41:51] when you want to do this next time, go to the partent task ->edit related tasks menu -> edit subtasks then add the subtask [19:42:14] it's not totally intuitive [19:42:23] ahh yeah [19:42:49] i tried to "add a subtask" but it just wanted to create a new one [19:48:25] AndyRussG: XenoRyet|brb antone coming to standup? [19:49:13] Xeno asked me to let yall know he'll be here in a minute [19:49:27] <--Cat :) [23:03:08] AndyRussG: of course! How could I confuse the optimistic futurism of the 60s with the apocalyptic futurism of the 70s?