[00:18:41] PROBLEM - Puppet run on tools-worker-1029 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:21:47] PROBLEM - Puppet run on tools-worker-1028 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:09:03] (03PS1) 10Krinkle: build: Fix broken autoload config [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342290 [01:28:17] (03PS1) 10Krinkle: Add 'debug' mode [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342291 [01:28:25] (03CR) 10Krinkle: [C: 032] build: Fix broken autoload config [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342290 (owner: 10Krinkle) [01:28:27] (03CR) 10Krinkle: [C: 032] Add 'debug' mode [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342291 (owner: 10Krinkle) [01:29:10] (03CR) 10jerkins-bot: [V: 04-1] Add 'debug' mode [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342291 (owner: 10Krinkle) [01:29:39] (03Merged) 10jenkins-bot: build: Fix broken autoload config [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342290 (owner: 10Krinkle) [01:29:41] (03CR) 10jerkins-bot: [V: 04-1] Add 'debug' mode [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342291 (owner: 10Krinkle) [01:29:47] (03PS2) 10Krinkle: Add 'debug' mode [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342291 [01:30:03] (03CR) 10Krinkle: [C: 032] Add 'debug' mode [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342291 (owner: 10Krinkle) [01:30:29] (03Merged) 10jenkins-bot: Add 'debug' mode [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342291 (owner: 10Krinkle) [01:33:28] (03PS1) 10Krinkle: Preserve 'debug' in the submitted form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342292 [01:34:01] (03CR) 10Krinkle: [C: 032] Preserve 'debug' in the submitted form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342292 (owner: 10Krinkle) [01:34:34] (03Merged) 10jenkins-bot: Preserve 'debug' in the submitted form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342292 (owner: 10Krinkle) [03:00:38] (03PS1) 10Krinkle: Start localisation of the GUC interface [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342298 (https://phabricator.wikimedia.org/T151657) [03:01:20] (03CR) 10jerkins-bot: [V: 04-1] Start localisation of the GUC interface [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342298 (https://phabricator.wikimedia.org/T151657) (owner: 10Krinkle) [03:02:15] (03PS2) 10Krinkle: Start localisation of the GUC interface [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342298 (https://phabricator.wikimedia.org/T151657) [03:02:49] (03PS3) 10Krinkle: Start localisation of the GUC interface [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342298 (https://phabricator.wikimedia.org/T151657) [03:03:59] (03CR) 10Krinkle: [C: 032] Start localisation of the GUC interface [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342298 (https://phabricator.wikimedia.org/T151657) (owner: 10Krinkle) [03:04:32] (03Merged) 10jenkins-bot: Start localisation of the GUC interface [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342298 (https://phabricator.wikimedia.org/T151657) (owner: 10Krinkle) [03:36:51] (03PS1) 10Krinkle: Preserve 'userlang' override when submitting the form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342299 [03:37:09] (03CR) 10Krinkle: [C: 032] Preserve 'userlang' override when submitting the form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342299 (owner: 10Krinkle) [03:37:18] (03CR) 10jerkins-bot: [V: 04-1] Preserve 'userlang' override when submitting the form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342299 (owner: 10Krinkle) [03:37:33] (03CR) 10jerkins-bot: [V: 04-1] Preserve 'userlang' override when submitting the form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342299 (owner: 10Krinkle) [03:38:34] (03PS2) 10Krinkle: Preserve 'userlang' override when submitting the form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342299 [03:40:05] (03CR) 10Krinkle: [C: 032] Preserve 'userlang' override when submitting the form [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342299 (owner: 10Krinkle) [04:09:01] (03PS1) 10Krinkle: Localisation for results-limited [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342300 (https://phabricator.wikimedia.org/T151657) [04:23:04] (03CR) 10Krinkle: [C: 032] Localisation for results-limited [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/342300 (https://phabricator.wikimedia.org/T151657) (owner: 10Krinkle) [04:44:06] madhuvishy: sorry but, which I can sudo, I just can't mess with other's files without explicit permission. [04:46:47] zhuyifei1999_: of course! Would you know who I can talk to to do that, and how I can reach them best? :) [04:47:15] well, it's matanya's files afaik [04:47:39] *which => while [04:48:42] zhuyifei1999_: okay [04:48:51] i'll ping them [04:57:09] PROBLEM - Free space - all mounts on tools-proxy-01 is CRITICAL: CRITICAL: tools.tools-proxy-01.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-proxy-01.diskspace.root.byte_percentfree (<33.33%) [05:47:08] RECOVERY - Free space - all mounts on tools-proxy-01 is OK: OK: tools.tools-proxy-01.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [06:45:28] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:25:27] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [08:39:47] 10Tool-Labs-tools-Other: wmflabs tool autodesc seems to have "disappeared", calls only return "Not Found" - https://phabricator.wikimedia.org/T160241#3092986 (10zhuyifei1999) a:05Cyberpower678>03Magnus [09:39:26] sql queries to labs DB replicas lags, my queries froze without result for tens of minutes [10:20:47] mbh: afaik, database replica lags and query freezing are two separate things [10:22:01] I call query freesing as "lags", but problem solved, my query was not optimised [10:23:08] https://tools.wmflabs.org/replag/ [12:45:04] !log utrs upgrade production/database software to latest versions [12:45:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [12:48:55] !log utrs install/configure mysql-server on database [12:48:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [12:50:37] !log utrs install apache2 on production [12:50:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [12:58:42] 10Tool-Labs-tools-Pageviews: add appropriate lang and dir attributes to the page title near the dates range under the chart - https://phabricator.wikimedia.org/T160247#3093106 (10Amire80) [12:58:59] 10Tool-Labs-tools-Pageviews, 07I18n, 07RTL: add appropriate lang and dir attributes to the page title near the dates range under the chart - https://phabricator.wikimedia.org/T160247#3093121 (10Amire80) [13:12:36] 10Tool-Labs-tools-Pageviews, 07I18n: Pageviews-num-pageviews works incorrectly in Hebrew - https://phabricator.wikimedia.org/T160248#3093124 (10Amire80) [13:44:12] 06Labs: Lost Wikitech 2FA details, recovery needed - https://phabricator.wikimedia.org/T159521#3093141 (10mschwarzer) @Aklapper Is there anyhow a way to proceed with this? [14:01:40] (03CR) 10Lokal Profil: "needs one more bit to handle the few datasets which contain a `WHERE`" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342198 (owner: 10Lokal Profil) [14:32:38] !log utrs install php on production [14:32:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL [14:36:54] can someone tell me the easiest way to transfer files between lab instances? i'm hitting brick walls using scp [14:44:17] AmandaNP: You can connect to both instances from the stepstone. Just do the scp from the stepstone [14:44:39] stepstone? [14:44:41] Ehm, Bastion host. [14:44:48] oh [14:45:16] $ scp host1:somefile host2:heretoofile [14:45:19] * AmandaNP tries [14:47:31] wait a minute... [14:48:56] nope that fails too [14:49:02] different error tho [14:49:17] Host key verification failed [14:50:29] You can ssh into that host without problems? [14:51:11] both of them yes [14:51:19] right from bastion [14:53:17] my previous issue was permission denied publickey [14:53:22] even though I was using -i [14:53:40] when I was just doing it right between the two instances [15:02:30] even if I add my key to authorized keys [15:20:17] screw all that, got sftp to work through filezilla. 10x easier [15:20:22] * AmandaNP gets to work [17:15:53] should I be able to enable a security group on an instance through horizon or do I need someone higher up to do so for me? [17:23:19] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Fix or delete user accounts showing on wikitech but not found in LDAP after T149109 - https://phabricator.wikimedia.org/T159986#3093268 (10Physikerwelt) @bd808 thank you. At least the user I was in contact with had already created a new account. [17:26:25] 06Labs, 10Labs-Infrastructure: Labs instance utrs-primary is running Ubuntu Precise and must be rebuilt. - https://phabricator.wikimedia.org/T159737#3093269 (10DeltaQuad) I apologize for the delay on this. It was scheduled a while back, but other critical errors and my work schedule prohibited this from happen... [17:28:35] AmandaNP: you should be able to do so in horizon [17:28:46] AmandaNP: but iirc it may take some time for it to come into effect [17:29:02] I can't find the option at all, unless creating one does it automatically [17:29:33] the create form should ask which secutiry groups to enable [17:29:52] create form for an instance? [17:30:51] cause it's already created [17:30:54] and when you go to Instances, then 'Edit instance' under 'Actions', you can change the groups [17:31:29] fml I just found it when you said that [17:31:37] that button should be more descriptive... [17:33:41] k, i'll wait a bit and try again later to see if it worked. [17:33:44] thanks valhallasw`cloud [18:03:47] 06Labs, 10Tool-Labs: Tool "autodesc" 404s - https://phabricator.wikimedia.org/T160255#3093271 (10Magnus) [18:18:17] AmandaNP: are you getting anywhere? [18:18:50] still waiting on that security group to apply [18:19:02] what instance and what security group? [18:19:21] it should only take like 5 minutes to update [18:19:25] utrs-database, database security group [18:20:21] so it's port 3306 you're needing open? [18:20:31] ya [18:20:36] wait [18:20:42] crap.. /me think [18:21:18] 3306 looks open to me [18:21:36] ya I just realized thats not the port... [18:21:55] or it is? [18:21:58] ugh [18:25:12] AmandaNP: I'll be in and out today but will keep a window open, feel free to hail me if I can do anything to help. [18:25:39] k. i'm trying to figure out if this is an application issue or a security group issue [18:30:52] andrewbogott: if i'm connecting from one instance to another, do I need both to have the security group? or just the inbound? [18:32:08] Just inbound, almost always [18:32:21] What I do to see if a port is blocked is [18:32:29] $telnet utrs-database.utrs.eqiad.wmflabs 3306 [18:32:44] If it immediately kicks me out then the port is open. If the port is firewalled then it will hang for a long time [18:33:04] (sorry, you maybe know this already, just suggesting a sanity check) [18:33:36] that helps. [18:33:48] so the port isn't blocked...which means something on my end [18:35:50] madhuvishy: looked for me? [18:36:04] I'm just getting "telnet: Unable to connect to remote host: Connection refused" [18:36:34] which is the same thing i'm getting with my web app [18:39:13] Silly question, is MySQL actually running on utrs-database and listening on that port? [18:40:06] mysql is running [18:40:10] checking listening [18:41:17] tcp 0 0 localhost:mysql *:* LISTEN [18:41:33] so i'd say yes stwalkerster [18:44:09] fairly sure that means it's only listening on it's localhost interface, not it's ethernet interface. [18:45:30] aww crap [18:49:35] stwalkerster: i've now set the IP and I have [18:49:36] tcp 0 0 utrs-database.utr:mysql *:* LISTEN [18:49:49] still getting refused [18:51:07] oh wait [18:51:28] now on my local machine I get access denied. /me looks into this [18:52:51] ok, production can connect, can the web interface do so [18:53:02] no. [19:11:33] matanya: Hi! Yes, can you look into deleting old files from the video project on labs. Its at 2tb usage right now, and we are at really high overall usage. This project is using 40% of all available nfs labs storage [20:54:58] 06Labs: Providing index of backlinks table to labs replicas - https://phabricator.wikimedia.org/T159984#3085420 (10Umherirrender) You have always to provide the namespace to make better database access (but I do not know, if the index is there) Try this: > SELECT COUNT(*) FROM pagelinks WHERE pl_namespace = 0... [21:22:59] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3093442 (10Aklapper) Could only be "fixed" when assuming that everybody else would share the same understanding of "slow" and everybody would run into the very same problem with any query. See https://mediawik... [21:25:29] madhuvishy: will do [21:31:05] 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3093447 (10IKhitron) Hi. Slow is just slow. Takes more time. I run every week dozens of queries. Each took less than 5 seconds in the past. It takes at least a couple of minutes each now. [21:34:57] 06Labs, 10Tool-Labs: Shut down "cewbot" - https://phabricator.wikimedia.org/T160264#3093452 (10MarcoAurelio) [22:08:21] 06Labs: Providing index of backlinks table to labs replicas - https://phabricator.wikimedia.org/T159984#3093478 (10Ebraminio) 05Open>03Resolved a:03Ebraminio Excellent, it makes a huge difference for my use. [23:07:50] Hi [23:08:24] I've a little question, any plan to support Java 8 on tools servers? [23:12:50] andrewbogott: getting a python error of [23:12:54] pywikibot.exceptions.NoUsername: Failed OAuth authentication for wikipedia:en: The authorization headers in your request are not valid: The request came from an invalid IP address. [23:12:54] [23:13:06] so I tried to reallocate the floating IP we got [23:13:14] it won't let me [23:13:24] which part won't it let you do? [23:13:24] keeps saying it's unable to [23:13:28] Releasing it? Or reassigning? [23:13:49] it doesn't say anything releasing it, but when I hit assign it hard fails [23:14:27] ok, looking... [23:15:14] we're talking about utrs-database? [23:16:02] it's supposed to be assigned to production [23:16:12] was assigned to primary before [23:17:04] ok… try now? [23:18:21] ok it assigned. [23:18:26] let me check the script [23:18:42] great. I don't know why it didn't release properly before, I just deleted the IP from the commandline [23:19:20] * AmandaNP reboots the instance to be sure it gets the ip after more failure on the script [23:21:33] still getting " The request came from an invalid IP address." [23:21:42] * AmandaNP trys to do more googling [23:23:08] AmandaNP: I don't know if somewhere your bot is permitted only with a specific IP… the newly assigned floating IP is different (208.80.155.169) from the IP you were using before (208.80.155.172) [23:23:17] So if something elsewhere is explicitly looking for .172 then this won't work [23:23:33] hmm /me opens up the bot [23:23:39] 10Tool-Labs-tools-Other: wmflabs tool autodesc seems to have "disappeared", calls only return "Not Found" - https://phabricator.wikimedia.org/T160241#3093500 (10Magnus) See https://phabricator.wikimedia.org/T160255 [23:23:42] But I don't have any idea what that 'invalid IP address' message really means [23:24:07] 06Labs, 10Tool-Labs: Tool "autodesc" 404s - https://phabricator.wikimedia.org/T160255#3093501 (10Magnus) Others reported this at https://phabricator.wikimedia.org/T160241 [23:26:41] it's set to allow all IPv4 and v6 [23:27:23] ugh I wish how I still had the docs from last time I set this up [23:28:42] that's the mwoauthdatastore-bad-source-ip message [23:29:11] that sounds like what it is...I just don't know what the problem or solution is [23:29:35] * Ensure the request comes from an approved IP address, if IP restriction has been [23:29:38] * setup by the Consumer. It throws an exception if IP address is invalid. [23:29:52] AmandaNP: I'd setup OAuth authentication again [23:30:10] Is it possible that the user auth is failing so it's failing back on an anonymous connection which has a more restrictive IP policy? [23:30:12] like on the instance? [23:31:46] Platonides: does that mean that AmandaNP is using a token that's bound to a particular host, so they need a new token to use the bot from a new host? [23:33:39] * andrewbogott needs to go for a bit but will check back before leaving for the night [23:40:53] WOW [23:41:14] that is one roundabout way to say httplib2 doesn't come with standard python [23:47:55] 06Labs, 10Tool-Labs: Tool "autodesc" 404s - https://phabricator.wikimedia.org/T160255#3093503 (10Peachey88) [23:47:57] 10Tool-Labs-tools-Other: wmflabs tool autodesc seems to have "disappeared", calls only return "Not Found" - https://phabricator.wikimedia.org/T160241#3093505 (10Peachey88) [23:48:42] andrewbogott: that's what it seemed [23:48:55] what I copied was the description of the function throwing the exception