[06:32:14] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/20after4 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=874627 edit summary: [06:54:06] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.009 second response time [07:10:01] hmm [07:10:11] * twentyafterfour doesn't know where tools.wmflabs.org is hosted, let me see... [07:16:41] horizon shows the following for the tools project: "error: unable to retrieve instances" [07:16:58] and "Error: Unable to retrieve usage information." ...etc [07:17:12] other projects are fine [07:19:04] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.024 second response time [07:46:59] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [08:50:29] twentyafterfour: shinken is flaky [09:35:35] (03CR) 10Lokal Profil: "> > IS it possible to use the harvesting in the bot development to" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [09:46:26] (03CR) 10Lokal Profil: "OK. When I try to follow the steps I get the following error when I get to step 3" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [10:18:57] (03CR) 10Lokal Profil: [C: 04-1] "> (1 comment)" (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [10:41:13] (03CR) 10Lokal Profil: [C: 04-1] "> > What was the reason for reverting the other patch?" (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313451 (owner: 10Jean-Frédéric) [10:43:45] (03CR) 10Lokal Profil: [C: 031] "I'm happy with the current solution but since the patch now has my fingerprints all over it I'll pass on the +2 baton to you." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [10:53:53] (03PS1) 10Lokal Profil: Updating .gitignore [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313755 [10:56:46] (03CR) 10Lokal Profil: Updating .gitignore (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313755 (owner: 10Lokal Profil) [10:58:35] twentyafterfour, you don't have access to Horizon in projects that you aren't project admin in. [11:14:14] (03PS4) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [11:15:27] (03CR) 10jenkins-bot: [V: 04-1] Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:46:44] (03PS5) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [11:46:57] (03CR) 10Lokal Profil: "Runs fine when I try it on de-he_(de) [after re-basing on, and using the steps in, Ib7a7aa56c15444c59c7299b42c0b38944b2a817e]" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:47:22] (03CR) 10jenkins-bot: [V: 04-1] Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [12:11:48] (03PS4) 10Lokal Profil: [BLOCKED] Store plain permalink instead of urlencoded one [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) [12:59:51] 10Wikibugs, 07Easy: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2682583 (10Vaishnav980123) a:03Vaishnav980123 I would like to work on this bug. [13:03:31] (03PS2) 10Lokal Profil: Extract CommonsCat templates data to JSON file [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309842 (owner: 10Jean-Frédéric) [13:04:47] (03CR) 10Lokal Profil: Extract CommonsCat templates data to JSON file (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309842 (owner: 10Jean-Frédéric) [13:05:47] (03PS1) 10Lokal Profil: Re-factor test of getCommonscatTemplates [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313760 (https://phabricator.wikimedia.org/T145574) [13:12:44] 10Wikibugs, 07Easy: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2682626 (10Vaishnav980123) a:05Vaishnav980123>03None [13:12:45] (03PS6) 10Lokal Profil: Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) [13:13:24] (03CR) 10jenkins-bot: [V: 04-1] Only output one primkey warning per page [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/312975 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [13:26:29] (03Draft1) 10Paladox: Change project Project-Creators to Project-Admins [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) [13:26:31] (03Draft2) 10Paladox: Change project Project-Creators to Project-Admins [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) [13:29:09] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#2682642 (10Vaishnav980123) a:03Vaishnav980123 I would like to work on this bug. [13:32:36] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Documentation: Create a "my first Pywikibot bot" tutorial for Tool Labs - https://phabricator.wikimedia.org/T134495#2682649 (10Vaishnav980123) a:05Vaishnav980123>03None [13:39:16] (03CR) 10Luke081515: [C: 031] Change project Project-Creators to Project-Admins [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) (owner: 10Paladox) [13:50:33] (03CR) 10Jean-Frédéric: [C: 032] Extract CommonsCat templates data to JSON file [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309842 (owner: 10Jean-Frédéric) [13:51:11] (03Merged) 10jenkins-bot: Extract CommonsCat templates data to JSON file [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309842 (owner: 10Jean-Frédéric) [13:52:27] (03CR) 10Jean-Frédéric: [C: 04-1] Re-factor test of getCommonscatTemplates (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313760 (https://phabricator.wikimedia.org/T145574) (owner: 10Lokal Profil) [13:53:39] (03CR) 10Jean-Frédéric: "> How about expanding the API dev bit in the README to say something" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [14:04:03] (03CR) 10Jean-Frédéric: "> > OK. When I try to follow the steps I get the following error when" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [14:06:35] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2682729 (10Rithika) a:03Rithika I would like to work on this bug [14:07:28] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2682737 (10Paladox) a:05Rithika>03None @Rithika please stop spamming. [14:11:28] (03CR) 10Jean-Frédéric: "Thanks André!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309842 (owner: 10Jean-Frédéric) [14:18:51] (03CR) 10Jean-Frédéric: Updating .gitignore (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313755 (owner: 10Lokal Profil) [14:19:55] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2548434 (10Niharika) >>! In T142851#2682737, @Paladox wrote: > @Rithika please stop spamming. @paladox I'd expect better behaviors from you towards new contributors. [14:21:38] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2682767 (10Paladox) @Niharika hi, sorry, but @Rithika has similar behaviour to @Vaishnav980123 Sorry if I mistakenly said they were spammers, and wont happen again. [14:24:14] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2682768 (10Niharika) >>! In T142851#2682767, @Paladox wrote: > @Niharika hi, sorry, but @Rithika has similar behaviour to @Vaishnav980123 > > Sorry if I mistakenly s... [14:33:53] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2682789 (10Paladox) Oh, sorry I didn't know, but it wont happen again. I thought they were spammers because One of them assigned this task then decided to let it go.... [15:17:08] Niharika hi, im sorry earlyer for mistakinly thinking those users were spamming [15:17:37] paladox: No worries. Assume Good Faith though! [15:18:23] Yep [15:52:26] Is 208.80.155.255 (internal-server-nat.wmflabs.org) the HTTP proxy for any project in Labs? [15:55:35] Niharika: just wondering, what's the point of claiming a task, then undo like very soon after, for multiple tasks? [15:58:29] zhuyifei1999_: None at all. The kids didn't know if they will be able to do a ticket before they claimed it and then when it seemed too tough, they gave it up. We're encouraging them to not do so now, so hopefully you'll see less of it. [16:06:08] k [16:07:05] can you encourage them to like "not give up"? many tasks here are tough [16:19:52] SPF|Cloud, no [16:20:04] it's not an HTTP proxy [16:21:01] Can you explain what it is? Somebody using that IP is flooding my site with 2 or 3 requests per second [16:21:51] It's the IP used by all labs instances which don't have their own public IPs [16:22:15] for any type of connection to the internet [16:22:30] understood [16:24:03] what exactly are they doing? [16:26:05] Flooding us with POST request [16:30:57] what URL are they trying, can you send me the user agent, etc.? [16:57:26] (03PS1) 10Jean-Frédéric: Break down monuments_config in individual JSON files [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313766 (https://phabricator.wikimedia.org/T145333) [17:01:34] e: comets got z-lined? [17:02:04] that's a quit message [17:03:45] a very misleading one, in that case [17:11:01] What you get when it's a free text box [17:15:35] he always quits with that message [17:16:20] <|L> a) there are no z-lines at freenode (they have another name), b) the message ocntains a "Quit:" so it's manually triggered [17:16:43] <|L> for example you can easily quit with "Quit: K-Lined", just do /quit K-Lined [18:03:03] 10Tool-Labs-tools-Erwin's-tools: For related changes make category changes switchable. - https://phabricator.wikimedia.org/T146594#2665801 (10Nemo_bis) How useful is the "hidewikidata" option? I'm tempted to remove it and just add `AND rc_type <= 4` after line 184 `WHERE sc_depth < ' . $d . '`. [18:18:51] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:18:58] |L, what's the equivalent of a Z line at freenode? [18:30:52] <|L> tom29739: D-Line [18:53:01] hey all! anyone here know about the uwsgi python webservice? [18:53:25] chasemp: Hi. :) Any chance you can help with the above? [18:53:55] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:54:52] has anyone done "lazy-apps" with that uwsgi? [19:09:25] ori hmu when you get back [19:15:07] * ori waves [19:16:16] a wild mhashemi has appeared [19:16:23] what's up? [19:16:41] hey man! [19:16:41] sooo we're getting a bit of classic connection-gone-away action [19:16:52] this time with a sqlalchemy twist [19:17:35] * ori liked "Python by the C side ", and wondered about the mysterious hat-note [19:18:08] what tool / labs host / log file are we looking at? [19:18:08] hehe I've done no C integration for this i assure you [19:18:14] montage-beta [19:18:23] and this all worked fine in montage-dev, same env [19:18:31] I'll show ya the logs in a mo [19:19:11] what's the project name? [19:19:12] now uwsgi is almost certainly configured to do preforking, and it's loading the app in the master/arbiter, meaning any module-level connections are fakakta in the workers [19:19:18] montage-beta :) [19:20:15] so i did everything i could to make everything lazy [19:20:34] from creating the engine on first request to using uwsgi lazy-app [19:21:01] nothin, even a 17-second-old worker can't db connect [19:21:13] http://tools.wmflabs.org/montage-beta/login is the repro [19:21:25] http://tools.wmflabs.org/montage-beta/meta/ has a bunch of info [19:21:48] i usually just use that to make sure my workers are actually cycling (they do, pretty much all the time, these days) [19:21:50] !bash so i did everything i could to make everything lazy [19:21:50] ori: Stored quip at https://tools.wmflabs.org/bash/quip/AVeG2GIFRMLREAjJrkhJ [19:22:18] haha that's legit what people recommend, right down to zzzzeek himself [19:22:53] bc yes i will reach out to the maintainer of even the most popular project when it gives me issues: https://twitter.com/mhashemi/status/782334698357829632 [19:23:20] watch me ask linus about kernel bugs in short order [19:23:43] I'm trying to add myself to the tool and failing [19:24:44] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=managemembers&projectname=tools&servicegroupname=tools.montage-beta&returnto=Special%3ANovaServiceGroup [19:24:52] "You must be a member of this service group to perform this action." [19:24:52] http://stackoverflow.com/questions/25904817/ningx-uwsgi-python-permanent-mysql-error-after-some-time-from-starting-applica [19:24:55] is like the exact issue [19:25:01] where this action == adding myself to the service group [19:26:36] am i really not on wikitech? is this not wm global id or whatever? [19:29:34] I'm not sure [19:29:45] can I live-hack the code? [19:29:47] is it versioned? [19:30:47] mhashemi: ^ [19:30:51] also, can you add me to the tool? [19:31:10] it is definitely versioned [19:31:18] make a branch, commit what's on there already, and go for it [19:31:30] and i'm tryna log in to this [19:32:09] like, are wikitech creds not the same as wikipedia creds? [19:32:18] no mhashemi [19:32:22] yeah, separate [19:33:31] your username on wikitech is MahmoudHashemi [19:34:25] http://tools.wmflabs.org/montage-beta/#/login works now? [19:34:26] welcome to the group tools.montage-beta, Ori.livneh [19:34:35] oh never mind [19:34:41] thanks [19:34:44] http://tools.wmflabs.org/montage-beta/login [19:34:46] yeah [19:34:52] close but no cigar [19:35:01] unless the cigar isn't a cigar, freudianly [19:35:13] and yeah, fixed my creds [19:39:44] PROBLEM - SSH on tools-webgrid-generic-1404 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:05] good times [19:41:34] mhashemi: OK, so I wanted to get to the bottom of the exception, rather than the error we were seeing, which seemed to emanate from a log handler [19:41:59] so I edited montage/server.py and changed line 151 from: [19:42:19] "if replay_log_path:" to "if False and replay_log_path:" [19:42:35] then I ran "webservice uwsgi-python restart" [19:42:42] and it seems to work now [19:43:02] hah, for real? let's see [19:43:52] ho boy, yeh [19:43:54] i see [19:45:12] perhaps the logging middleware contains some magic that runs after the database connection is closed, or something like that [19:45:44] that's a shot in the dark, since I'm not familiar with this stack at all [19:46:08] nah nah this is like, same symptom, two causes, and you definitely got the right cause :) [19:46:38] i addressed the log handler btw [19:46:55] i need to undo some of that laziness i think [19:47:42] ok :) [19:48:30] ori: mhashemi: shall I tell folks to run with the tool? :D [19:48:55] yeah dude, let's unleash em, if they haven't fallen asleep [19:49:00] i'll update the status page [19:49:15] (https://commons.wikimedia.org/wiki/Commons:Montage/Status) [19:49:19] India is asleep, Latvia is not. [19:49:25] thank to both of you, mhashemi and ori. [19:49:46] a very special thanks to you, ori. :) [19:50:08] I really didn't do anything -- it was wild luck [19:50:27] but hopefully it stays on your side! [19:50:51] ttyl, happy rosh hashana [19:51:34] status page updated, and happy rosh hashana to you and all, thank you very much! [19:54:34] hi [19:56:15] mhashemi: ori leila just got back, all good? [19:56:57] ( yuvipanda: in a call, will be back in 15 min or so) [20:05:53] mhashemi: ori I see you aren't setting pool_recycle in the create_engine call, without that this will continue happening with sqlalchemy and mysql I think [20:06:12] if you aren't creating a new mysql connection per request (which is what I usually end up doing) [20:11:19] mhashemi: ori https://github.com/hatnote/montage/pull/2 should fix it [20:13:33] ah yes, that was one of the first things i tried, it's on the server, but not in the github, sorry! [20:13:33] ori: mhashemi I think it was fixed rn only because restarting reset all the connections [20:13:35] i set it to 60 too [20:13:44] and no, it wasn't that i'm afraid [20:14:01] restarting decidedly didn't fix the connections, super frustrating! [20:14:15] because i was actually using a connection to verify that the maintainers had their accounts at startup [20:15:47] mhashemi: ah, ok. is it all sorted out now, or is the cause still unknown? [20:16:51] mhashemi: also is this connecting to tools.labsdb or one of the replica hosts? [20:22:37] mhashemi: I'm going to fall asleep shortly. am in Barcelona rn. leila should be able to give you contact info for me :) (usual phone number / telegram) [20:23:42] yuvipanda: sleep well, and thank you for coming in such a short notice. [20:23:51] * leila waves to yuvipanda [20:24:20] good night! [20:49:15] PROBLEM - Puppet staleness on tools-worker-1018 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [22:10:59] (03PS2) 10Jean-Frédéric: Updating .gitignore [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313755 (owner: 10Lokal Profil) [22:11:51] (03CR) 10Jean-Frédéric: Updating .gitignore (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313755 (owner: 10Lokal Profil)