[04:38:23] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: tools.merlbot stopped working - https://phabricator.wikimedia.org/T135006#2296242 (10bd808) @Merl, @Luke081515 Is the bot functioning again? [04:42:47] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace.root.byte_percentfree (<22.22%) [07:06:18] !log deployment-prep rebooting deployment-tin (denying access again) [07:06:31] wm-bot: ? [07:06:31] Hi mobrovac, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [07:06:44] *sigh* [08:34:53] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:47:41] Change on 12www.mediawiki.org a page OAuth (obsolete info)/en was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2133637 edit summary: Updating to match new version of source page [08:47:42] Change on 12www.mediawiki.org a page OAuth (obsolete info)/ja was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2133638 edit summary: Updating to match new version of source page [09:34:59] 10Wikibugs: Don't echo on IRC when title change only - https://phabricator.wikimedia.org/T135363#2296523 (10Danny_B) [12:29:26] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: tools.merlbot stopped working - https://phabricator.wikimedia.org/T135006#2296931 (10Luke081515) 05Open>03Resolved Yep. Thank you very much for that fix :) [12:46:38] ggfggg [12:47:23] YuviPanda: ? [13:00:51] !log ores deploying e51c4f3 to celery and web nodes [13:00:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [13:21:52] 10Tool-Labs-tools-Global-user-contributions, 07Easy, 07JavaScript: GUC project total count doesn't increment - https://phabricator.wikimedia.org/T123879#2297031 (10Krinkle) [13:21:54] 10Tool-Labs-tools-Global-user-contributions: Tool no longer shows number of projects with edits - https://phabricator.wikimedia.org/T118662#2297034 (10Krinkle) [13:24:07] 10Tool-Labs-tools-Global-user-contributions: Tool no longer shows number of projects with edits - https://phabricator.wikimedia.org/T118662#2297036 (10Krinkle) [13:29:43] 06Labs: Investigate labnet1002 kernel panic - https://phabricator.wikimedia.org/T135322#2297054 (10chasemp) p:05Triage>03High a:03Andrew [13:31:07] 06Labs: Investigate labnet1002 kernel panic - https://phabricator.wikimedia.org/T135322#2295290 (10chasemp) @andrew I'm tossing this your way, iirc we had similar issues nearly a year ago prompting our labnet host failover testing. Is there still known kernel updates pending here? Thoughts on upgrading? [14:13:27] Change on 12www.mediawiki.org a page OAuth/ca was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137647 edit summary: Updating to match new version of source page [14:13:28] Change on 12www.mediawiki.org a page OAuth/de was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137648 edit summary: Updating to match new version of source page [14:13:28] Change on 12www.mediawiki.org a page OAuth/en was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137649 edit summary: Updating to match new version of source page [14:13:29] Change on 12www.mediawiki.org a page OAuth/es was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137650 edit summary: Updating to match new version of source page [14:13:29] Change on 12www.mediawiki.org a page OAuth/fr was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137651 edit summary: Updating to match new version of source page [14:13:30] Change on 12www.mediawiki.org a page OAuth/it was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137652 edit summary: Updating to match new version of source page [14:13:30] Change on 12www.mediawiki.org a page OAuth/ja was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137653 edit summary: Updating to match new version of source page [14:13:31] Change on 12www.mediawiki.org a page OAuth/ms was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137654 edit summary: Updating to match new version of source page [14:13:31] Change on 12www.mediawiki.org a page OAuth/pl was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137655 edit summary: Updating to match new version of source page [14:13:32] Change on 12www.mediawiki.org a page OAuth/pt was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137656 edit summary: Updating to match new version of source page [14:13:32] Change on 12www.mediawiki.org a page OAuth/pt-br was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2137657 edit summary: Updating to match new version of source page [14:38:23] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2297238 (10jcrespo) This is not a #DBA ticket, I made sure that replica was working and safe (filtered) literally hours after the wiki was setup: ``` $ mysql -... [14:41:05] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2297245 (10Krenair) >>! In T135029#2297238, @jcrespo wrote: > I suppose some cron/script maintenance on #labs may be failing to execute? I'm not aware of anythin... [14:59:06] 10Tool-Labs-tools-Global-user-contributions, 07Easy, 07JavaScript: GUC project total count doesn't increment - https://phabricator.wikimedia.org/T118662#2297335 (10Danny_B) p:05Triage>03Low a:03Devirk [15:25:19] 06Labs: image rsync from labcontrol1001 to labcontrol1002 never cleans up - https://phabricator.wikimedia.org/T134769#2297450 (10Andrew) 05Open>03Resolved [15:26:47] 10Tool-Labs-tools-Other: Some geolocated photos not shown - https://phabricator.wikimedia.org/T105593#2297451 (10simon04) WIWOSM queries https://tools.wmflabs.org/geocommons/kml?bbox=... for images in the current map view. @para, does this request return objects with `{{location}]` but not `{{object location}}`? [15:31:41] 10Tool-Labs-tools-Global-user-contributions, 07Easy: GUC project total count doesn't increment - https://phabricator.wikimedia.org/T118662#2297488 (10Krinkle) "count doesn't increment" inaccurately summarises the problem and makes assumptions about what the underlying bug is. [15:33:30] 10Tool-Labs-tools-Global-user-contributions, 07Easy, 07Regression: GUC tool always outputs "searched 1 projects" instead of actual count - https://phabricator.wikimedia.org/T118662#2297495 (10Krinkle) [15:41:36] 06Labs: confirm that new base labs base image is adequate for kubernetes &c. - https://phabricator.wikimedia.org/T134944#2297532 (10yuvipanda) p:05Triage>03Normal [16:02:25] 10PAWS: Write 2-3 year vision for annual plan - https://phabricator.wikimedia.org/T131267#2297649 (10yuvipanda) p:05Triage>03Normal [16:20:18] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2286195 (10chasemp) Can someone update this task description? As-is I have no idea what's going on here. I know this is in large part because of my own lack of... [16:26:15] 06Labs: Investigate labnet1002 kernel panic - https://phabricator.wikimedia.org/T135322#2297760 (10Andrew) https://phabricator.wikimedia.org/P3109 [16:40:37] 06Labs, 10DBA: Replicate CN related tables at labs - https://phabricator.wikimedia.org/T135405#2297843 (10Steinsplitter) [16:41:35] 06Labs, 10DBA: Replicate CN related tables at labs - https://phabricator.wikimedia.org/T135405#2297856 (10Steinsplitter) [16:48:54] 10Tool-Labs-tools-Other, 10Analytics, 06Community-Tech, 10Pageviews-API, and 2 others: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2297886 (10Nuria) Seems that this is already done: https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&start=2016-... [16:49:13] 10Tool-Labs-tools-Other, 10Analytics, 06Community-Tech, 10Pageviews-API, and 2 others: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2297887 (10Nuria) 05Open>03Resolved [17:14:43] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2297959 (10debt) Hi there - we need to update the stats on the wikipedia.org portal page on a regular basis. We've recently found that the script we run to do th... [17:17:49] 10Tool-Labs-tools-Global-user-contributions, 07Easy, 07Regression: GUC tool always outputs "searched 1 projects" instead of actual count - https://phabricator.wikimedia.org/T118662#2297972 (10Danny_B) @krinkle Thanks for title clarification. Why did you remove #javascript actually? [17:19:37] 06Labs: Investigate labnet1002 kernel panic - https://phabricator.wikimedia.org/T135322#2297981 (10Andrew) a:05Andrew>03MoritzMuehlenhoff Moritz, can you comment on the above stack trace? Does it look familiar/like a known kernel issue? [17:20:25] 10Tool-Labs-tools-Global-user-contributions, 07Easy, 07Regression: GUC tool always outputs "searched 1 projects" instead of actual count - https://phabricator.wikimedia.org/T118662#1806012 (10Glaisher) It's generated on the PHP side. [18:15:47] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2298231 (10chasemp) [18:17:37] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2298268 (10MaxSem) [18:19:06] MaxSem: why are you reverting chasemp? [18:19:30] ? [18:19:50] ah [18:20:21] either it's bad info (which is from their product manager?) or idk [18:20:33] that ticket has a lot going on [18:20:54] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2298285 (10MaxSem) I've edited the description back. The issue in this ticket is unavailability of adywiki replicas. The updates are discussed in a blocked ticke... [18:21:25] valhallasw`cloud, chasemp ^ [18:21:33] MaxSem: thanks for the clarification. [18:22:33] so, what's going on with these? [18:22:45] we have the replicas but not the views? [18:23:01] that's one part of it it seems yes [18:23:18] this ticket really needs some details https://phabricator.wikimedia.org/T135029 [18:23:26] I didn't know that ws a new wiki etc [18:23:35] hard to walk into that task w/o doing a lot of my own digging [18:23:49] but in essence, wiki setup seems to be badly documented historically for labsdb [18:24:08] and no one around now may have ever done it [18:24:08] so further complications [18:24:28] chasemp: someone must've done it for jamwiki [18:25:04] YuviPanda, O RLY? $ sql jamwiki_p [18:25:05] ERROR 1049 (42000): Unknown database 'jamwiki_p' [18:25:12] are you sure? [18:25:18] yeah I think they are both in a similar state [18:25:20] yeah, just double checked [18:25:22] kk [18:26:05] MaxSem: why does this job run in labs and not say analytics for the portal? [18:26:13] avail of the replica aside this all seems very strange [18:26:42] chasemp, because portals get rebuilt on developers' machines [18:26:56] so it makes no sense to export these stats from prod [18:27:08] if labs (used to) work just fine [18:31:11] I need to dash to the office, be back in an hour [18:53:26] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: No replica for adywiki - https://phabricator.wikimedia.org/T135029#2298442 (10chasemp) >>! In T135029#2297238, @jcrespo wrote: > This is not a #DBA ticket, I made sure that replica was working and safe (filtered) literally hours... [18:59:17] is wikiscan on labs servers? [19:00:46] !log wikiscan is it [19:00:47] wikiscan is not a valid project. [19:01:01] ToAruShiroiNeko: what's the URL? [19:01:05] if it has one [19:01:16] sure [19:01:22] http://wikiscan.org [19:01:42] It feels independent [19:01:46] ToAruShiroiNeko: no, that does not look like it's on labs [19:01:56] the IP resolves to something on poneytelecom.eu [19:07:27] ok thanks [19:59:57] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: setup Letsencrypt for toolserver.org (toolserver.org certificate to expire 2016-06-30) - https://phabricator.wikimedia.org/T134798#2298927 (10Dzahn) [20:00:07] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: setup Letsencrypt for toolserver.org (toolserver.org certificate to expire 2016-06-30) - https://phabricator.wikimedia.org/T134798#2277377 (10Dzahn) p:05Triage>03Normal [20:09:38] Can you create temporary tables using quarry? [20:10:23] tom29739: what about make a try? :D [20:10:27] YuviPanda: ^ [20:10:57] 'Access denied for user 'u2029'@'%' to database 'u2204__testing'' [20:11:10] 'create temporary table if NOT EXISTS u2204__testing.pageid ( id int(11) PRIMARY KEY );' [20:11:18] That's my sql query. [20:11:30] I guess I have to run it on labs directly then. [20:12:00] why not? :D [20:19:00] Because quarry is easier and it's easier to publish the results. [20:28:09] tom29739: no, because you can't write into other people's tables [20:29:43] That makes sense. [20:29:44] hm, so we have to take the user account data from quarry, in theory [20:30:04] you could, but I'd reccomend not doing it :) [20:30:19] until we get more hardware for labsdb [21:04:12] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2299063 (10RobH) Please note that this allocation, via procurement task T131871, has been approved. Two of our spare systems wmf4657 & wmf465... [21:07:47] 'ERROR 1436 (HY000) at line 54: Thread stack overrun: 72048 bytes used of a 196608 byte stack, and 128000 bytes needed. Use 'mysqld --thread_stack=#' to specify a bigger stack. [21:07:47] ' [21:09:04] I get that when I try to run my sql query, but when I use the option '--thread_stack=200048', it says 'mysql: unknown variable 'thread_stack=200048' [21:09:05] ' [21:09:15] And the mysqld command is not found. [21:09:22] I'm confused. [21:14:22] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2299085 (10EBernhardson) I'm thinking it will be simpler to give them a service cluster name, makes things easier to remember. If there are no... [21:15:25] tom29739: looks like an error from the server, not from the client [21:15:47] How do I fix it? [21:15:57] I don't understand mysql [21:16:46] are you doing anything with stored procedures? [21:17:16] tom29739, what's your query? sounds like it's hitting some mysql's internal limits [21:17:17] or it could be this bug: https://stackoverflow.com/questions/21344167/why-is-a-one-line-mysql-stored-function-overflowing-the-stack [21:17:21] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: setup Letsencrypt for toolserver.org (toolserver.org certificate to expire 2016-06-30) - https://phabricator.wikimedia.org/T134798#2299093 (10Dzahn) >>! In T134798#2294608, @yuvipanda wrote: > It's in relic.toolserver-legacy.eqiad.wmflabs I can't logi... [21:18:25] valhallasw`cloud, MaxSem, it's this: https://phabricator.stwalkerster.co.uk/P11 [21:18:37] stwalkerster made it, I'm trying to use it. [21:19:33] infinite recursion? [21:19:49] That could happen. [21:20:45] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: setup Letsencrypt for toolserver.org (toolserver.org certificate to expire 2016-06-30) - https://phabricator.wikimedia.org/T134798#2299101 (10Dzahn) please disregard. pebcak. using bastion.wmflabs and not bastion-restricted.wmflabs was the issue [21:20:46] the max recursion depth might trigger it, but I'm not sure. In general, I'd check with jynus whether it's safe to run this on the replicas at all [21:22:33] Why wouldn't it be safe? [21:22:50] It might kill replication, or cause unreasonable load on the server. [21:23:16] It seems to work on a category without any subcategories. [21:23:37] And on this category: https://en.wikipedia.org/wiki/Category:User_en [21:23:52] I'll ask before I start it again. [21:56:31] 06Labs, 10Labs-Infrastructure, 06Operations: Get labs-ns0, labs-recursor0, and labservices1001 on the same system, and labs-ns1, labs-recursor1, and holmium on another - https://phabricator.wikimedia.org/T135447#2299210 (10Andrew) [21:59:54] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Get labs-ns0, labs-recursor0, and labservices1001 on the same system, and labs-ns1, labs-recursor1, and holmium on another - https://phabricator.wikimedia.org/T135447#2299236 (10Andrew) Deployment plan: 1) disable puppet on holmium and on la... [22:29:20] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: setup Letsencrypt for toolserver.org (toolserver.org certificate to expire 2016-06-30) - https://phabricator.wikimedia.org/T134798#2299300 (10Dzahn) {F4021144} [22:33:24] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: setup Letsencrypt for toolserver.org (toolserver.org certificate to expire 2016-06-30) - https://phabricator.wikimedia.org/T134798#2299314 (10Dzahn) works now after the fixes above :) all details were in P3110 [22:33:37] tom29739: yeah, that query is completely broken, there's no way that'd work on anything like the larger categories, plus it doesn't take into account loops in the category graph [22:33:37] 06Labs, 10Tool-Labs, 06Operations, 13Patch-For-Review: setup Letsencrypt for toolserver.org (toolserver.org certificate to expire 2016-06-30) - https://phabricator.wikimedia.org/T134798#2299315 (10Dzahn) 05Open>03Resolved [22:34:30] It works on ones with a small amount of subcategories. [22:34:53] I'll make a better script sometime. [22:35:12] In a language I can actually understand in some detail :D [22:35:14] tom29739: yeah, what that was attempting to do though was somewhat insane [22:35:27] Yep. [22:44:49] 06Labs, 06Discovery, 10Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#1386072 (10Dzahn) I just noticed there are still a lot of hits for these on the toolserver-legacy instance, there are requests for a.toolserver, b.tools... [22:54:34] 06Labs, 10Labs-Infrastructure: Redirect https://toolserver.org/~magnus/ - https://phabricator.wikimedia.org/T113696#1673644 (10Dzahn) but https://tools.wmflabs.org/magnustools/thetalkpage is also 404 so that would not change much. there are a lot of existing redirects for other magnus tools already, but not... [22:55:49] 06Labs, 10Labs-Infrastructure: Redirect https://toolserver.org/~magnus/ - https://phabricator.wikimedia.org/T113696#2299370 (10Dzahn) fwiw, the existing rules are in : puppet://modules/toolserver_legacy/templates/www.toolserver.org.erb [23:06:35] 06Labs, 10Tool-Labs: Make http (404, 302, 301 etc) statistics for toolserver.org - https://phabricator.wikimedia.org/T85167#940446 (10Dzahn) list of tools that still generate 404s (as of access.log of today, which starts about 17h ago), so just today; toolserver.org/~adler toolserver.org/~agony toolserver.org... [23:14:46] 06Labs, 10Tool-Labs: Make http (404, 302, 301 etc) statistics for toolserver.org - https://phabricator.wikimedia.org/T85167#2299443 (10Dzahn) by FAR the highest number of hits is still for "cmarqu". it's more than everything else combined and then some... [23:24:18] 06Labs, 10Tool-Labs: Make http (404, 302, 301 etc) statistics for toolserver.org - https://phabricator.wikimedia.org/T85167#2299464 (10Dzahn) | tool | num hits | adler |2 | agony |2 | aka |24 | al |17024 | alchimista |4 | alexz |3 | amidaniel |2 | animist |2 | annabel |2 | apper |21 | authoritycontrol |16 | a... [23:25:25] 06Labs, 10Labs-Infrastructure, 07Need-volunteer: Redirect https://toolserver.org/~magnus/ - https://phabricator.wikimedia.org/T113696#2299466 (10Dzahn) [23:26:06] 06Labs, 10Labs-Infrastructure, 07Need-volunteer: Redirect https://toolserver.org/~magnus/ - https://phabricator.wikimedia.org/T113696#1673644 (10Dzahn) also see T85167#2299464 [23:58:44] hi, any labs admin with some free time around? I need some help with https://phabricator.wikimedia.org/T131630