[00:23:04] (03PS1) 10Andrew Bogott: Added dummy passwords for labsdnsconfig [labs/private] - 10https://gerrit.wikimedia.org/r/273149 [00:25:13] (03CR) 10Andrew Bogott: [C: 032 V: 032] Added dummy passwords for labsdnsconfig [labs/private] - 10https://gerrit.wikimedia.org/r/273149 (owner: 10Andrew Bogott) [00:28:45] (03PS1) 10Andrew Bogott: One-letter typo makes puppet compiler unhappy [labs/private] - 10https://gerrit.wikimedia.org/r/273150 [00:28:54] (03CR) 10Andrew Bogott: [C: 032 V: 032] One-letter typo makes puppet compiler unhappy [labs/private] - 10https://gerrit.wikimedia.org/r/273150 (owner: 10Andrew Bogott) [00:35:01] Would/can someone from wikimedia-labs provide a copy of the now offline http://scribunto.wmflabs.org/index.php/Module:TestUtils . It is mentioned at https://test2.wikipedia.org/wiki/Help:Lua_development_environment as able to provide a 'toframe' method which would be very helpful for developing some lua code from the command line, instead of having to do all of the editing through a textarea. [00:40:26] cariaso: I don [00:40:30] 'grr [00:40:46] I don't think that instance was archived anywhere [00:41:19] Have you looked around on mediawiki.org to see if there is something similar? [00:42:26] cariaso: this may be of interest -- https://www.mediawiki.org/wiki/Module:UnitTests [00:43:13] bd808 thanks, seems like a solid start. [00:43:44] And https://en.wikipedia.org/wiki/Wikipedia:Lua#Unit_testing [00:44:01] oh the module is on enwiki -- https://en.wikipedia.org/wiki/Module:UnitTests [00:44:15] my goal is to be able to do some linux command line dev/test of lua scripts that will deploy into my mediawiki. any other pointers or resources that help this are of great interest. [00:45:09] the usage at the top of enwiki is particularly helpful [00:46:02] cariaso: if you can find anomie when he is active (US EST timezone) he maybe be able to point you to many things. He worked on Scribunto and wrote a lot of Lua scripts along the way [00:47:20] cariaso: you can find a lot of scripts using https://en.wikipedia.org/w/index.php?title=Special%3APrefixIndex&namespace=828 [00:53:00] cariaso: there are also a lot of tests with the scribunto extension source -- https://phabricator.wikimedia.org/diffusion/ELUA/browse/master/tests/ [00:53:26] great tip, thanks [02:17:17] RECOVERY - Puppet failure on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [02:34:20] PROBLEM - Puppet staleness on tools-worker-1004 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [43200.0] [03:42:07] 6Labs, 10Tool-Labs: tools-bastion-05 is super slow - https://phabricator.wikimedia.org/T127992#2062419 (10MZMcBride) >>! In T127992#2060909, @Multichill wrote: > I noticed @marcmiquel running big jobs on the bastion the other day that ran on 95%+ CPU for hours. Is there any way to prevent this type of resourc... [06:27:42] 6Labs, 10Tool-Labs: tools-bastion-05 is super slow - https://phabricator.wikimedia.org/T127992#2062550 (10bd808) >>! In T127992#2062419, @MZMcBride wrote: > Is there any way to prevent this type of resource hogging more strictly? Whether the underlying excessive resource usage is malicious or accidental, when... [07:38:15] bd808: toolserver had a 500MB quota, so doing 40GB NFS things was simply impossible :-) [07:46:37] 6Labs, 10Tool-Labs, 10DBA: What would be the preferred way to run a series of read queries on DB replica, each on two *_p databases ? - https://phabricator.wikimedia.org/T124805#1966479 (10jcrespo) I do not know if you are looking for advice of for permission. I cannot give you too much advice, but regarding... [09:07:57] 10Quarry: Allow users to choose free licenses other than CC0 - https://phabricator.wikimedia.org/T124528#2062669 (10yuvipanda) 5Open>3declined I don't really know the answer to the question of copy pasting stuff from fawiki to here. However, that problem will be present no matter what individual license is p... [11:24:34] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:27] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [11:40:41] 6Labs, 10wikitech.wikimedia.org: Need to reset two-factor authentication for wikitech account - https://phabricator.wikimedia.org/T127999#2062913 (10Krenair) a:3Krenair Yes. I will deal with the verification for this later today. [13:59:50] 6Labs, 10Tool-Labs-tools-Other, 6Developer-Relations: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#2063219 (10Qgil) A step forward? {T122865} [14:26:11] mornin' valhallasw`cloud :) (or appropriate greeting) [14:44:23] chasemp: *waves* [14:48:27] Hi [14:48:37] I'm setting up a project named algo-news [14:48:54] To run it in a normal way I need Celery [14:49:14] On Wikipedia there is jstart etc. [14:49:58] fako: that's not enough information to be helpful on our end, can you write up a more complete summary of what you want to do and why you cannot? [14:50:44] Celery is a job queue for Python [14:51:17] I'm doing a lot of processing in the background, but everything uses Celery [14:51:37] So I need to either rewrite it to use Sun Grid Engine [14:51:39] https://pypi.python.org/pypi/gridmap [14:51:46] With something like that [14:52:02] Or I spawn a Celery process that will handle the jobs that I send it. [14:52:13] Celery could for instance run under jstart [14:52:26] Is that more clear? [14:52:31] that is probably going to be complicated [14:52:54] when you say you are starting up a new project in our parlance it means specifically you are setting up a cluster of VPS's which you will manage [14:53:04] if you mean you are writing a new tool for the Tools env [14:53:13] Yes, that's what I mean [14:53:15] I'm not sure that will be a configuration you are happy with [14:53:18] It's already there btw [14:53:27] but I honestly just don't know [14:53:39] best to persist to a task and we can see if anyone is doing something similar? [14:53:43] it's not an unreasonable idea [14:53:52] but it's also not a simple overlay for celery on grid [14:54:12] fako: https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ [14:56:54] Shall I ask there if launching a Celery through jstart is a good idea? [14:57:42] Fako: I think there are tools using celery workers via a redis queue [14:57:48] How does this help in "seeing if anyone is doing something similar?" [14:58:14] Yes, a Redis queue would be fine [14:58:37] Question is how to run Celery or is there a Celery running on the platform somewhere that is shared by these tools? [15:03:44] Maybe if you can give me the name of such tools I can take a peak myself? [15:12:10] fako: you can just start celery workers on the grid? [15:13:03] fako: crosswatch [15:13:14] Haven't tried yet. Will give it a shot. [15:27:02] 6Labs, 6Operations, 10ops-codfw: Figure out what labstore hardware is viable in codfw - https://phabricator.wikimedia.org/T128083#2063474 (10chasemp) [15:27:52] I'm not allowed to view crosswatch (ls in the directory gives perm. denied) [15:30:40] 6Labs, 6Operations, 10ops-codfw: Figure out what labstore hardware is viable in codfw - https://phabricator.wikimedia.org/T128083#2063519 (10Papaul) @chasemp yes all those boxes are still in place but nerve got configured. see ticket below T102626 [15:30:41] I do think I will get Celery to run inside jstart [15:31:29] It runs, but it tries to connect to the wrong queue and over SSL atm and gives errors [15:36:30] hi guys! [15:36:30] quick question, is there any way to obtain info regarding the editors - their bots? if it would be scalable to several languages it would be even better [16:02:27] bd808: Good news, my config issue is resolved. Thanks for your help :) [16:02:52] I will add the result of this to my documentation if I got time :) [16:03:00] Luke081515: awesome. obviously we need some better docs on the config system [16:25:31] PROBLEM - SSH on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:44] marmick: no -- mediawiki doesn't map bots to their users (although you might be able to do something with the links on the bots' user page? if it redirects, it typically redirects to the owner) [16:36:46] valhallasw`cloud: nice idea. i didn't think of that :) thanks [16:36:54] valhallasw`cloud: would there be a record for bot approval? [16:37:04] is that a common practice? [16:37:23] chasemp: there might be a log entry 'X granted the bot flag', but I'm not sure if that would mention the user [16:37:38] table user groups [16:37:41] says which bots are approved [16:37:47] but it says nothing about 'the owner' [16:38:03] sometimes you can guess because they are composed: valhallasw`cloud-bot [16:38:08] hm yeah [16:38:12] but other times it's valhallabot [16:38:18] I've only gotten one bot approved on wiki and I frankly can't remember [16:38:19] which completely invalidates a method [16:38:34] well, it doesn't completely invalidate it -- it just means the heuristic doesn't work for all usernames [16:38:41] that's inherent with heuristics [16:39:00] yes, well, you get a smaller population [16:39:15] perhaps a pairing of 5% of all existing bots [16:40:23] RECOVERY - SSH on tools-webgrid-lighttpd-1204 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [16:45:51] marmick: so you want to tie on wiki human to on wiki bots? [16:45:55] is that the end game? [16:46:14] :) to understand better how humans engage with wikipedia [16:46:18] I would ask in a more wiki admin oriented channel if anyone has ideas [16:46:37] chasemp: yes [16:46:40] I can't mentally bridge that gap atm but there has to be a way [16:46:51] as far as available data [16:46:56] there's only mediawiki and dumps [16:47:16] do you suggest any channel? [16:50:48] #wikipedia-en? [16:50:53] or whichever wiki you're looking at [16:51:13] each wiki has different rules when it comes to bots, so each wiki might need a different heuristic [16:52:21] i see [16:52:22] good point [16:53:44] You can look in the account creation log to see who created a bot account, but I guess only a small number of people created their bot accounts while being logged in [16:54:22] I would expect most, if not all, bot accounts to be created while logged in. [16:55:30] And I remember a big discussion about this on the analytics list. See https://lists.wikimedia.org/pipermail/analytics/2016-January/thread.html#4858 WikimediaBot convention [16:55:38] thanks [16:56:03] they are ideas to work on [16:59:30] marmick: Most wiki's require bots to be linked to the operator usually with a {{bot}} template. That should give enough info to link the two together [17:00:43] ok, amybe different strategies can give a better result [17:04:16] multichill: your a good dude thanks [17:06:08] you're welcome :-) [17:16:02] Krenair: I think most people don't even know you can do that -- you have to manually create the 'create an account' url. I didn't know it was possible until two years ago someone mentioned it to me [17:16:16] Krenair: 'create an account' is obvious when you're logged out, not so when you're logged in [17:27:16] 6Labs, 10wikitech.wikimedia.org: Need to reset two-factor authentication for wikitech account - https://phabricator.wikimedia.org/T127999#2063916 (10Krenair) 5Open>3Resolved [17:48:31] Thanks valhallasw`cloud and chasemp I think I'll get it working. [17:56:15] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2064062 (10kaldari) [18:15:44] 10PAWS: "pwb.py login" causes password request - https://phabricator.wikimedia.org/T128108#2064170 (10sumanah) [18:17:43] 10PAWS: "pwb.py login" causes password request - https://phabricator.wikimedia.org/T128108#2064189 (10sumanah) According to the manual: "Pywikibot needs a configuration file 'user-config.py' which PAWS automatically creates for you, with https://test.wikipedia.org/ as the default wiki." Just now: ``` Sumanah@P... [18:21:20] bd808: Do you know which settings if have to use for $wgSMTP, if I want to let my wiki send mails? [18:21:42] 10PAWS: "pwb.py login" causes password request - https://phabricator.wikimedia.org/T128108#2064232 (10sumanah) p:5Triage>3High I am calling this high priority because I know @jayvdb is teaching some workshops in the next few days that will use PAWS, and because this is a showstopper for testing that the user... [18:25:43] (03PS1) 10Matanya: fix typo [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273285 [18:26:02] (03CR) 10Matanya: [C: 032 V: 032] fix typo [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273285 (owner: 10Matanya) [18:31:00] (03CR) 10ArthurPSmith: "FYI I don't quite have the next set of changes ready - I was trying to implement resizing the page which was another request there - I'm l" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/271551 (owner: 10ArthurPSmith) [18:31:26] (03PS1) 10Matanya: current stewards web page on tools [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273288 [18:31:29] Luke081515: mediawiki-vagrant installs a mail handler that traps all mail on the VM, delivered to the "vagrant" user. [18:31:46] (03CR) 10Matanya: [C: 032 V: 032] current stewards web page on tools [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273288 (owner: 10Matanya) [18:31:55] so there are no easy steps to enabling outbound mail at the moment [18:32:45] :-/ [18:32:46] I can't remember if we have a phab task open to make it possible to change that or not [18:33:05] https://phabricator.wikimedia.org/T117391 ? [18:33:08] this is all really designed to be a testing platform [18:33:38] ok [18:33:42] Luke081515: *nod* that would be the same thing [18:33:52] I think using internal mail is enough [19:00:52] (03PS1) 10Matanya: update stewards page on tool labs [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273294 [19:01:13] (03CR) 10Matanya: [C: 032 V: 032] update stewards page on tool labs [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273294 (owner: 10Matanya) [19:04:27] (03PS1) 10Matanya: no toolserver email [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273295 [19:04:43] (03CR) 10Matanya: [C: 032 V: 032] no toolserver email [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273295 (owner: 10Matanya) [19:07:30] (03PS1) 10Matanya: fix link [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273297 [19:08:44] (03CR) 10Matanya: [C: 032 V: 032] fix link [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/273297 (owner: 10Matanya) [19:32:03] hello, I need a labs admin, someone here? [19:32:53] bd808: Are you able to force restart instances? [19:33:23] Luke081515: 'force restart'? [19:33:29] you can just do that in wikitech yourself? [19:33:33] sorry, I mean hard reboot [19:33:45] oh, seems like problem self solved [19:33:52] the instance hanged completly, but rebootet now [19:34:36] Luke081515: the 'reboot' button in wikitech is a hard reboot [19:35:00] is it? I was not sure [19:35:18] before that, I used the hard reboot button at horizon, but this is now broken :-/ [19:35:31] those buttons are the same thing afaik [19:36:07] but horizon nows hard reboot and normal reboot for exmaple [19:36:23] and at wikitech you can't shutoff a instance via webUI I think [19:38:05] no, you can't, because you also can't turn it on again. [19:38:27] at horizon this was possible [19:38:36] some time ago [19:38:54] but since https://phabricator.wikimedia.org/T127440 this is broken [19:42:20] Luke081515: well, wikitech is the supported ui, and horizon is the 'we hope to switch to this in the future' ui, so if it's not in wikitech, consider it non-existing [19:42:34] hm, ok [19:43:19] bd808: What I have to do to load https://gerrit.wikimedia.org/r/#/c/273293/ ? git-update didn't help, I got still the same error as at T128102 [19:43:20] T128102: Can't use role liquidthreads and role massmessage at vagrant - https://phabricator.wikimedia.org/T128102 [19:43:31] what a service :D [19:44:01] I think you have to manually git pull in /srv/mediawiki-vagrant and run `vagrant provision` [19:48:18] 6Labs, 10DBA: Measure capacity and utilization of labsdb*** boxes - https://phabricator.wikimedia.org/T107070#2064513 (10jcrespo) p:5Low>3High [19:48:45] legoktm: Thanks for that tip & for your fast patch, it works now [19:49:03] awesome [19:49:54] 6Labs, 6Operations, 10ops-eqiad: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2027631 (10jcrespo) Cmjohnson: See my comment on T118174#2062707 [20:13:33] 6Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2064562 (10Andrew) @scfc: I don't think it's related to the outage, but those queries are very strange! I see lots like that whenever I turn on logging; I'll try to see where they are coming from. [20:14:35] 6Labs, 13Patch-For-Review: pdns trying to resolve wikimedia.org.eqiad.wmflabs - https://phabricator.wikimedia.org/T128123#2064566 (10Andrew) [20:23:14] 6Labs: pdns trying to resolve wikimedia.org.eqiad.wmflabs - https://phabricator.wikimedia.org/T128123#2064599 (10Krenair) [20:24:07] andrewbogott, that's interesting. do instances attempt those lookups when they fail to find the domain elsewhere? [20:24:44] no, that wouldn't make sense.. [20:24:54] Krenair: It’s possible… it could just be noise from some external badly behaved client as well [20:25:06] do you know instances are trying to resolve what domains? [20:25:20] So far I haven’t been able to log who is making those queires [20:25:24] Coren, wsexport is dead, many export in sleep state since yesterday, can you restart it? [20:25:25] queries [20:25:56] phe: can you tell me more about what needs doing? is wsexport a tool that just needs a ‘webservice restart’? [20:26:17] okay, do you know which exact domains they're looking for? [20:27:37] If, from a labs box, I do ‘dig labcontrol2001.wikimedia.org.eqiad.wmflabs’ I see roughly the same thing in the logs [20:28:01] andrewbogott, terminate all task with user = wsexport on tools-webgrid-lighttpd-1404 and restart the webservice [20:28:51] sure but are they all looking up labcontrol2001 in that manner? [20:28:51] phe: that’s all managed by the grid, right? why would I need to log on to tools-webgrid-lighttpd-1404 specifically? [20:28:55] or another system? [20:29:01] Krenair: no, it’s various things. [20:29:04] including labcontrol2001 [20:29:05] and virt0 [20:29:08] and *.wikimedia.org [20:29:11] none of it makes sense really [20:29:12] virt0? [20:29:13] andrewbogott, right, you don't need it [20:29:14] huh [20:29:31] Krenair: hang on, I’ll show you what I’m looking at [20:29:34] yeah, would be good to know what instances are trying it then, I guess [20:30:22] !log tools.wsexport running webservice restart [20:30:31] phe: any better? [20:30:36] thanks [20:31:19] andrewbogott, yes, it start to answer request [20:31:37] !log tools.wsexport running webservice restart [20:31:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wsexport/SAL, dummy [20:32:48] 6Labs: pdns trying to resolve wikimedia.org.eqiad.wmflabs - https://phabricator.wikimedia.org/T128123#2064617 (10Andrew) https://phabricator.wikimedia.org/P2674 [20:32:52] Krenair: https://phabricator.wikimedia.org/P2674 [20:34:44] andrewbogott, looking at the timestamps, probably all connected to a single issue [20:35:02] Krenair: what do you mean? [20:35:15] it seems to happen in bursts [20:35:28] ton of broken-looking queries in a second, then it stops for a few seconds [20:36:29] oh, yeah — well, even a single dig produces 6 queries in the log. [20:36:42] so each ‘burst’ is just one query I think [20:37:07] the first broken query in each group - labcontrol2001, then virt1000, then virt0 [20:37:12] yes [20:37:24] so perhaps... [20:37:33] let me try hitting pdns directly without the recursor [20:38:05] same behavior. It’s not the recursor embellishing [20:38:07] and yes, it certainly appears that each group is one single query [20:38:09] 10PAWS: "pwb.py login" causes password request - https://phabricator.wikimedia.org/T128108#2064629 (10sumanah) CLIENT_ID isn't a key set in os.environ, so it can't use OAuth: ``` >>> for k in os.environ.keys():... [20:38:53] is virt1000 still around? [20:39:43] no, you got rid of all the ^virt.* machines right? [20:39:55] correct [20:40:04] there's now ^labvirt100[0-9] - and they're in .eqiad.wmnet instead [20:40:17] and ^labvirt101[0-1] [20:40:30] the new ‘labvirt1000’ is ‘labcontrol1001' [20:41:36] and labcontrol1002 is..? [20:42:02] 10PAWS: os.environ doesn't get client ID, etc. necessary for PAWS session to use OAuth - https://phabricator.wikimedia.org/T128108#2064649 (10sumanah) [20:42:57] labcontrol1002 is a spare for labcontrol1001 [20:43:00] ok [20:43:04] labcontrol2001, on the other hand, no longer exists. [20:43:18] right [20:47:55] 6Labs, 10Labs-Infrastructure, 10DBA: Get HA db support for labs services - https://phabricator.wikimedia.org/T126251#2064689 (10jcrespo) [20:51:57] andrewbogott or other Tool Labs admins: Feel like killing and disabling jobs by tools.rezabot? It has broken code that's making bad login attempts at a rate of several per second, and has been for weeks despite the operator being pinged multiple times on various wikis. [20:52:21] andrewbogott, I'm wondering if instances with broken puppet are behind this [20:52:56] any progress on labsdb1002 (c2.labsdb)...? I'm waiting for it [20:53:09] Krenair: check this out: http://blog.powerdns.com/2014/02/06/related-to-recent-dos-attacks-recursor-configuration-file-guidance/ [20:53:22] (not related to the weird queries, but possibly related to the outages) [20:53:28] anomie: I think tools.rezabot is a different bot [20:53:40] liangent: what are you waiting for, specifically? I think all dbs should still be available [20:53:57] anomie: I’ll have a look... [20:54:20] andrewbogott: I use user dbs [20:54:26] andrewbogott, bd808: Well, whatever it is is trying to log in as user Rezabot. [20:54:38] although I have back up of db contents from the original host [20:54:50] if I restore it to the current temporary host [20:54:59] I think it will be lost again when you switch it back [20:55:15] anomie: but fawiki's https://fa.wikipedia.org/wiki/%DA%A9%D8%A7%D8%B1%D8%A8%D8%B1:Rezabot points to Yamaha5 and tools.rezabot is https://fa.wikipedia.org/wiki/%DA%A9%D8%A7%D8%B1%D8%A8%D8%B1:Reza1615 [20:55:32] liangent: I don’t know anything outside of https://phabricator.wikimedia.org/T126946 [20:55:42] anomie: I'm going to make a config patch so we can get the real ip [20:56:25] bd808: IP isn't really a problem, it's tool labs IPs. And logging into each one shows at least one process running out of /data/project/rezabot/ [20:56:31] there are quite a lot of active jobs in tools.rezabot [20:56:40] andrewbogott: or, will there be any prior announcement before getting the fixed c2 back online [20:57:02] liangent: I don’t know. Probably /when/ it is fixed there will be an announcement [20:58:00] and there are a ton of cron entries… lots going on in tools.rezabot [20:58:13] (it's pretty obvious when doing a `tail -f api.log | grep action=login` on fluorine) [20:59:45] anomie: Any idea which of these need killing? [20:59:51] https://www.irccloud.com/pastebin/dKGbi804/ [21:00:50] * liangent is not sure whether starting to work on c[13] is a good idea.. [21:02:13] andrewbogott: I'm looking for the ips [21:02:30] bd808: won’t they just all be ‘tool labs proxy’? [21:02:53] andrewbogott: no, labs->prod uses internal ips [21:03:02] ah, right [21:03:05] and most (if not all) exec hosts have an external ip as well [21:03:06] andrewbogott: They are like 10.68.17.* [21:03:37] andrewbogott: I can't really tell, but I'm seeing login attempts for Rezabot from 10.68.17.147, 10.68.17.252, 10.68.17.253, and 10.68.17.255. [21:04:00] yeah same list for me [21:04:03] so probably ‘everything’ :) [21:04:05] which is tools-exec-1210, -1213, -1214, and -1216 [21:04:25] I have a meeting shortly — if one of you wants to correlate those to job names and IDs I will kill on my return [21:04:55] andrewbogott: will do [21:06:36] andrewbogott: -1210 has 3813521 "zamandara" and 3808244 "rademaker_RC", -1213 has 3811634 "otherwiki", -1214 has 3782104 "neededarticle", and -1216 has 3792970 "purge_cat". [21:07:39] where does pywikibot keep it's cookie jar? [21:07:50] Hi all. [21:08:03] o/ dschwen [21:08:24] Can anyone point me to docs on how to access labsdb servers (commonswiki.labsdb for example) in a robust way from an arbitrary labs instance? [21:08:50] a while ago I copied the /etc/hosts and some iptables config from tool.labs to my instance [21:08:59] unsurprisingly that broke about a week ago [21:11:06] (03PS1) 10Youni Verciti: Add the public_html folder to easily update the html code [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/273320 [21:14:30] we used to have /data/project/.system/iptables.conf [21:14:34] that seems to be gone [21:14:58] dschwen: hmmm... robust is the key here certainly. [21:15:28] I know there is puppet code that sets up the host mappings for the tools project somewhere [21:15:34] dschwen: aiui *.labsdb is in dns these days [21:15:50] ok, good [21:16:00] so I'll clean my hosts out [21:16:20] bd808: re:cookie jar: pywikibot.lwp in the same directory user-config.py is loaded from [21:16:38] valhallasw`cloud: thanks! [21:18:15] ok, it is in DNS, that it great [21:18:23] I've flushed my iptables as well [21:18:37] and.. ta daa, it works [21:18:40] thx [21:18:43] anomie: ok, I’m back, just a moment... [21:20:58] anomie: I’m killing those jobs and commenting out the crons that start them [21:21:08] can you related what I’ve done to the tool admin and cc: me please? [21:21:45] andrewbogott: I'll post on their wiki page about it. [21:22:02] I'm going to find and nuke the pytwikibot cookie jars there [21:22:05] * anomie sees in api.log that Rezabot seems to have stopped [21:22:11] s/t// [21:22:27] !log tools.rezabot disabled quite a lot of tools and crons, as per anomie’s request: "It has broken code that's making bad login attempts at a rate of several per second, and has been for weeks despite the operator being pinged multiple times on various wikis." [21:22:51] labs-morebots: hello? [21:22:51] I am a logbot running on tools-exec-1221. [21:22:51] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [21:22:51] To log a message, type !log . [21:22:56] hmmmmm [21:23:04] !log tools.rezabot Deleted pytwikibot cookie jar files ./.pywikibot/pywikibot.lwp ./pycore/pywikibot.lwp [21:23:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.rezabot/SAL, Master [21:23:16] andrewbogott: it doesn't echo the first time [21:23:24] it's a bug in adminbot [21:23:31] hm, ok [21:23:34] it logged, in any case [21:23:51] andrewbogott: https://phabricator.wikimedia.org/T127410 [21:23:55] anomie: did I kill all that needed killing? [21:24:19] * andrewbogott often fixes morebots but would not like to be /in charge/ of fixing morebots [21:24:30] andrewbogott: For now anyway. Next week I might ask you to do another tool [21:24:38] 's jobs if that bot's operator doesn't respond. [21:24:42] heh. Yuvi just wants me to finish making it obsolete with stashbot [21:24:47] andrewbogott: BTW, https://www.wikidata.org/wiki/User_talk:Yamaha5#Please_attend_to_your_bot [21:25:22] anomie: thanks [21:35:05] andrewbogott, bd808: Ha, look at the right edge of https://grafana.wikimedia.org/dashboard/db/authentication-metrics?panelId=13&fullscreen (: [21:35:12] where do thumbnail error logs end up on beta? [21:35:26] all thumbnails are broken but I don't get anything in exception.log [21:35:26] *plop* [21:36:22] occasionally broken? they just started working again [21:43:46] anomie: nice [21:44:41] tgr: if they come from mw01 or mw02 then they go to logstash. if some other random code builds the thumbs then I have no idea [21:45:07] well, it started working again, so moot point [21:45:32] I suppose it was something with beta's cranky handwritten thumb.php [21:57:39] 6Labs, 10Tool-Labs, 13Patch-For-Review: Create a utility that lists all databases of a user - https://phabricator.wikimedia.org/T91231#2064909 (10scfc) 5Open>3Resolved [21:58:03] 6Labs, 10Tool-Labs: Create a utility that lists all databases of a user - https://phabricator.wikimedia.org/T91231#1077206 (10scfc) a:3scfc [22:50:11] 6Labs, 10Tool-Labs, 10Mail, 6Operations: remove toolserver mail aliases - https://phabricator.wikimedia.org/T127543#2046186 (10Dzahn) Hey @Yuvipanda should we delete that ? ts-admins@wikimedia ? [22:54:53] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [22:57:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:28:56] 10PAWS: os.environ doesn't get client ID, etc. necessary for PAWS session to use OAuth - https://phabricator.wikimedia.org/T128108#2065265 (10sumanah) @jayvdb just said to me: > Isnt PAWS using pywikibot 2.0? Which doesnt support oauth. I'm not sure what version it's using (and can't check at the moment). [23:34:57] RECOVERY - Puppet failure on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0]