[01:20:36] 6Labs, 10wikitech.wikimedia.org: Reset 2FA on wikitech for User:Revi and User:Hym411 - https://phabricator.wikimedia.org/T130994#2153593 (10revi) [[https://wikitech.wikimedia.org/wiki/Special:Log/Hym411|Special:Logs]]. [01:52:36] 6Labs, 10wikitech.wikimedia.org: Reset 2FA on wikitech for User:Revi and User:Hym411 - https://phabricator.wikimedia.org/T130994#2153599 (10JanZerebecki) 5Open>3Resolved a:3JanZerebecki [02:09:22] Niharika, ping [02:52:41] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2149727 (10Dereckson) I'm also affected by this issue and would like to see 2FA removed. [02:56:19] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2153616 (10JanZerebecki) @Dereckson correctly responded to my challenge, proving that he has access to the labs shell account dereckson [03:14:32] 6Labs, 10Horizon, 13Patch-For-Review, 7Tracking: Increase horizon session length - https://phabricator.wikimedia.org/T130621#2153622 (10Andrew) If I use SESSION_TIMEOUT values less than two hours, the setting has the desired effect. So someone is capping the value or there's an overflow or something. Cur... [04:23:34] Cyberpower678: Pong. [04:24:25] Niharika, I made some modifications to the checkIfDead class, but it would seem that timeouts are flagged as not dead sites. Or any site that has a curl error for that matter. [04:26:28] Cyberpower678: Oh. Good catch. It returns false in those cases? [04:26:34] Yep [04:26:56] Did a test using the multibatch function. [04:27:14] I gave the batch queries a 30 second timeout to account for bandwidth restrictions. [04:27:51] Cyberpower678: I should switch the current function to send false when site is dead and true when it's not then. That'll fix it. [04:28:27] Let me know in the pull request if you do. I'll have to update Cyberbot's code. [04:28:56] Cyberpower678: I will. That sounds good to you right? [04:29:08] Niharika, Btw, did the application submission go through yesterday? [04:29:24] Niharika, sure [04:29:36] Cyberpower678: The job application? Yep, I see yours. [04:29:56] Niharika, :-) [04:30:06] :) [04:33:32] Niharika, but we should also keep in mind other curl errors exist. When I tested the function which happened to have a nice sample of 143 urls executing simultaneously, only one timed out, but I saw SSL errors for some of them. [04:33:52] Though I'm not sure if that meant the curl itself failed or not. [04:34:24] But if the trues and falses get switched, I suppose that makes this concern moot. [04:35:29] Ah. [04:36:19] Niharika, alternatively, the default return value could be true. :P [04:36:36] Then you wouldn't need to switch them around. [04:38:08] Cyberpower678: Yeah, we can do that instead. Good idea. What modifications did you make to the class? [04:38:41] I added a new function. [04:39:05] It allows for simultaneous querying of any number of URLs. [04:39:14] Rather than do them in succession. [04:40:03] Cyberpower678: Okay. Wanna add a test for it? [04:40:41] * Cyberpower678 is notorius for testing in the debugger instead. :p [04:41:33] Niharika, I am unfamiliar with PHPUnit_Framework_TestCase atm [04:41:47] It's a pretty simple function. [04:42:11] Aha. Okay. I'll look it over and add one if I can. [04:42:26] You input an array of urls and the output is in the form of array( 'results'=>array(), 'errors'=>() ) [04:47:49] Cyberpower678: Does the table on logging interface still look meshed up to you? [04:48:01] 6Labs, 10Horizon, 13Patch-For-Review, 7Tracking: Increase horizon session length - https://phabricator.wikimedia.org/T130621#2153634 (10Andrew) Opened upstream bug https://bugs.launchpad.net/django-openstack-auth/+bug/1562452 And, proposed fix: https://review.openstack.org/#/c/298002/ [04:48:19] Niharika, yes [04:48:35] It's all pushed up against the right side of the page [04:48:41] Cyberpower678: Meshed up, as in? The date wraps to the next line - are you talking about that? [04:48:53] Cyberpower678: Okay, it's fine for me. Which browser? [04:49:11] Chrome 200% 4k resolution [04:49:29] And the date and Bot wrap to the next line [04:49:45] Some articles do too [04:51:09] Cyberpower678: Yeah, because they're too long. However it doesn't push up against the right side of the page for me. [04:51:18] Cyberpower678: Maybe try another browser? [04:52:16] Pushes to the right when the window is maximized [04:52:35] The table centers when I make the window small enough./ [04:52:44] For both browsers [04:53:03] Cyberpower678: Also, I want to ask you about the entire deadlink fixing process. Cyberbot fixed 351 links yesterday. Which comes to about ~4 minutes per link fix. It seems too slow. What's the reason behind this? [04:53:15] Cyberpower678: Okay. Let me try to replicate that issue. [04:53:45] Niharika, Cyberbot has pretty muched fixed all the links it could in the pages that have tags in them. [04:54:18] So it's pretty much looping through runs looking for the new tags essentially. [04:54:35] I have started the bot on the entire wiki to get the rest of the links cached. [04:54:55] But I have a second worker focusing on the pages only with tags on them. [04:54:59] Cyberpower678: But it's also checking for dead links itself now, right? Not just relying on tags? [04:55:06] Yeah, okay. [04:55:28] Niharika, not yet. It's functionality requires approval on enwiki. [04:55:33] Cyberpower678: So it's slow because it's checking links which have already been fixed? [04:55:34] A BRFA is open there. [04:55:59] Essentially. It's run is only 9 hours long. [04:56:17] It gets through the entire list of pages containing dead tags in roughly 9 hours. [04:56:28] Cyberpower678: Okay. That makes sense. Is there also a bandwidth issue? [04:57:01] Not at the moment. For the majority of those articles, Cyberbot has that information cached in the DB. [04:57:21] So it's literally plowing through those articles previously checked. [04:57:33] Cyberpower678: I see. In the long term though, moving it tool labs makes sense? [04:57:57] Niharika, that last part of your sentence is missing a vital word. [04:58:09] Cyberpower678: Which is? [04:58:17] I don't know. [04:58:28] I'm not sure what you're asking. [04:58:50] Cyberpower678: Cyberbot runs on your own server at the moment, right? [04:58:51] moving it from? moving it to? [04:59:10] Niharika, nope [04:59:17] It's running on tool labs [04:59:26] Cyberpower678: Oh, okay. I was under the impression that was the case. [04:59:30] My mistake. [04:59:46] I run multithreaded versions on my computer though, when benchmarking. [05:00:04] Cyberpower678: Should we spawn a few more bots to speed up the process? [05:00:19] Niharika, speaking of, using 20 workers, and assuming a complete DB, I pulled up to 40 articles per second. [05:00:51] Wow. It'll be awesome if we can get that speed into production. [05:01:10] Niharika, Labs staff has no intention of installing pthreads however. [05:01:27] To get those speeds, we need the pthreads extension. [05:01:46] Cyberpower678: But we can still make new bots, which do the same thing, right? [05:02:07] Sure, but they would start where the other bot left off. [05:02:20] Cyberbot is programmed to spawn in the middle of a list of articles. [05:02:28] *isn't [05:02:54] Cyberpower678: Yeah. We have an output API from the logger. Will that help? [05:03:09] Especially since it can only pull 5000 at a time, and it would need some kind of resume string to pass to the API to get it started in the middle. [05:03:28] Cyberpower678: See here: https://meta.wikimedia.org/wiki/Fixing_dead_links/Deadlink_logging_app [05:04:02] Niharika, The problem is that spawning multiple bots is pointless unless they start in different places on Wikipedia. [05:04:11] Niharika, how would you implement that? [05:04:52] Niharika, they would need to be distanced far enough from each other when it comes to call the API. [05:05:36] And to do that we would need to know how to call for a list of articles at a certain point in that list without initially using the continue parameter. [05:05:55] But then again the API is very big, so I might be missing a feature. [05:06:17] Cyberpower678: Hmm...There can be several possible solutions. Using categories, or wikiprojects, or just alphabetically. Say Bot1 check articles starting with A-K and Bot2 L-Z. [05:06:44] Niharika, I don't think the API sorts alphabetically though. [05:06:53] Lemme check. [05:08:17] Niharika, apprefix may work [05:08:30] I didn't know that existed. [05:09:08] We could spawn 26 bots that each work on a letter in the alphabet [05:09:48] That would require quite a bit of memory though. [05:10:26] Cyberpower678: We can have half of them on a new tool maybe? [05:10:43] Niharika, or we finally create the Cyberbot project. [05:10:59] Cyberpower678: Cyberbot project, as in? [05:11:04] And configure our own environment there. [05:11:14] Create a cyberbot project on labs [05:13:08] Cyberpower678: Let's do that then? [05:13:50] Cyberpower678: Apprefix works nicely: https://en.wikipedia.org/w/api.php?action=query&list=allpages&apprefix=H&aplimit=5 [05:19:30] 6Labs, 10Labs-Other-Projects, 10The-Wikipedia-Library: Create Cyberbot Project on Labs - https://phabricator.wikimedia.org/T112881#2153639 (10Cyberpower678) 5stalled>3Open p:5Low>3Normal [05:23:08] 6Labs, 10Labs-Other-Projects, 10The-Wikipedia-Library: Create Cyberbot Project on Labs - https://phabricator.wikimedia.org/T112881#2153642 (10Cyberpower678) So resource usage has been improved significantly. Average usage without the checkIfDead class, is roughly 30MB. Anticipating memory usage when the ch... [05:25:34] Niharika, I requested a 64GB project on labs. [05:26:08] Cyberpower678: I saw that. I want to echo a comment in there - why not use dumps/db instead of the API? It'd be faster, won't it? [05:26:29] If they approve we can make use of 64GB on labs. [05:26:37] Niharika, Not really. [05:27:14] Cyberpower678: Why not? API is the middleman. Directly accessing the DB would speed us up by a lot. [05:27:16] We need the page contents, and I fear the dumps are not updated frequently enough. [05:27:38] The replication DBs doesn't have the text table. [05:27:40] Cyberpower678: Aren't they updated hourly or something? [05:27:46] Which contains the page content. [05:28:04] They are. I thought they were update monthly or weekly. [05:30:07] Cyberbot fetches a list of 5000 articles at a time, which hardly takes any time. [05:30:37] It does a binary search on the API to find out when links were added at some point and saves it. [05:30:55] It fetches the page text by using action=raw on index.php [05:31:06] That's literally all it does. [05:31:16] Cyberpower678: Instead of directly jumping to spawning 26 bots, lets start with 2-3. That should reduce our RAM usage by a bit. 64 GB still seems a lot. [05:31:28] We wouldn [05:31:51] 't be using 64 GB right away, but I thought we wanted Cyberbot to go global at some point. [05:31:57] I'm factoring that in too. [05:32:10] Cyberpower678: " It does a binary search on the API to find out when links were added at some point and saves it." -- Why does it need to know when the links were added? And why save the information? [05:32:39] Niharika, to get a valid archive./ [05:32:40] Cyberpower678: Can't we get more RAM when we need it afterwards? [05:32:59] Niharika, I honestly don't know how labs is structured. [05:33:01] Cyberpower678: Valid archive? Isn't that what the deadlinks logger is for? [05:33:33] Cyberpower678: Okay, then let's ask for lesser RAM on the ticket if its possible to get more in the future, if its okay? [05:33:38] Niharika, Cyberbot retains the timestamp of when the link was initially added to an article. [05:33:48] Cyberpower678: Why is that important? [05:33:49] It uses that to request an archive made around that time. [05:34:02] Cyberpower678: Ah, okay but why save it? [05:34:15] So it doesn't have to run the binary search again in the future./ [05:34:49] Cyberpower678: Okay. [05:35:10] The binary search on the API is the most time consuming aspect when communicating with the API. [05:35:33] Cyberpower678: Yeah, I didn't realize this was happening. [05:35:54] So using the dumps, which are HUGE, is a bit unnecessary. [05:36:09] Cyberpower678: So let's ask for fewer resources right now and possibly expansion in the future when we need it? [05:36:15] Sure. [05:36:24] Want me to comment on the ticket? [05:36:40] I anticipate 28 bots will take at 5600 at the very max. [05:37:04] Like that's assuming the bot blows up memory when the checkIfDead multibatch query is used. [05:37:19] Which is dumped immediately after usage. [05:37:37] I'm thinking it will 2GB for 28 bots though. [05:38:00] Cyberpower678: Right. How much is offered by default? [05:38:18] 32-64GB I think. [05:38:57] Cyberpower678: That's perfect then, right? We can get our multiple bots up and running in a couple of days. [05:39:05] Indeed. [05:39:29] I just need to program the bots to be able to start at specific points in the API when told to. [05:40:09] I calculate the initial run would take around 5 months to finish. [05:40:19] After than much less. [05:40:29] calculate->estimate [05:41:24] Well I'm going to go to bed now. [05:43:46] Cyberpower678: Okay, could you comment on the ticket on how much RAM do we need finally and that if they will be able to upgrade us when we need it? [05:45:08] 6Labs, 10Labs-Other-Projects, 10The-Wikipedia-Library: Create Cyberbot Project on Labs - https://phabricator.wikimedia.org/T112881#2153643 (10Cyberpower678) Unless we can be upgraded in the future, we'll take 8GB to start with. That should be more than sufficient for now. [05:45:17] Niharika, ^ [05:45:49] Cyberpower678: Your sentence has a grammatical error. ;) [05:46:08] Or not. [05:46:11] Niharika, whaaaattt? [05:46:35] Cyberpower678: Nah, it's fine. I read it wrong. Goodnight! [05:46:58] Good night. Hope I hear back regarding the application soon. :-) [05:47:13] Yeah. [06:05:41] hello [06:06:03] hey guys anybody there [08:34:53] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [11:20:33] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:57:07] (03PS1) 10Youni Verciti: Rev 0.2.1 with documentation 0.2 [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/279750 [14:37:41] (03PS1) 10Youni Verciti: Rev 0.2.2 Add tools for fr:wv namespaces [labs/tools/vocabulary-index] - 10https://gerrit.wikimedia.org/r/279753 [15:01:23] Krinkle: Are you here? [15:28:32] Niharika, did you do anything to the table? [15:28:39] It's perfect now. :-) [15:29:19] 6Labs, 10Labs-Infrastructure, 7Graphite: Graphite is unable to detect, if a paused instance is resumed - https://phabricator.wikimedia.org/T131022#2153858 (10Luke081515) [15:29:24] Niharika, but Total pages processed seems to be stuck at 100 [15:29:54] Cyberpower678: No, I couldn't reproduce the issue at all. The bot is logging the revision ID as 0 for several records. Can you check what's causing that? [15:30:05] Cyberpower678: Yeah, I need to correct that. [15:30:12] That means the API failed to return a revisionID [15:30:31] Cyberpower678: How come that happened? Timeout? [15:31:12] It probably means the article wasn't changed. This could the result of a nobots, or the bot is disabled. [15:31:41] If someone disables the bot, the edit function returns false. [15:31:55] Same thing if there's a nobots template on the article. [15:32:05] False translates to 0 [15:32:37] Cyberpower678: But it still returns a number for number of links fixed. That should be 0 as well in that case. [15:32:44] Thinking about it, that's more of a feature than a bug. It tells which pages the bot tried to edit, but couldn't [15:32:47] Cyberpower678: Or that case should not be logged at all. [15:33:22] Niharika, you could think of it as an attempted edit that failed. [15:33:59] With stats containing Cyberbot's intentions. You could easily filter out the entries with a revid of 0 though. [15:34:02] Cyberpower678: But the number of links fixed is misleading. It should be number of links not fixed, if anything. It obviously did not fix anything. [15:34:39] Niharika, Cyberbot is able to fix them though. It's just not able to edit the page. [15:35:10] Cyberpower678: I don't understand. It's fixing the link but not editing the page? How is that possible? [15:35:50] Like I said, something occured during the editing process that caused the edit function to return a false instead of a revision ID. [15:36:06] A nobots could do that, or the API failed at that very moment. [15:36:39] So the edit was successful? [15:37:07] If it got a false response, the edit was unsuccessful. [15:37:13] Most likely. [15:37:18] yuvipanda: are you available? [15:38:30] Niharika, I can program the bot to internally output unsuccessful edit attempts./ [15:39:01] Cyberpower678: If the edit fails, the best option would be to retry the edit immediately. [15:39:07] And give up after X tries. [15:39:39] Niharika, I've actually encountered situations where that is rather harmful./ [15:40:05] How come? [15:40:49] Once when the API returns a 503, the edit actually still went through, but my bot didn't receive a valid response so it retried 20 times. Well the bot ended up tagging an article 20 times, and really spammed the talk page as a result. [15:41:04] *returned [15:42:03] Cyberpower678 How come the API returned 503 20 times? What could cause that? [15:42:26] Niharika, I think there was some maintanance issues going at that time. [15:43:00] But the bot was retrying 20 times on every page, and made a real mess of things as a result./ [15:43:30] Cyberpower678: We can retry 3 times, and if it doesn't work, we give up. Actually, I can add an additional API parameter so, if its a failed edit, you mark that as true and we can highlight it in red on the logger. [15:43:30] 10Quarry: Quarry stuck on "queued" state - https://phabricator.wikimedia.org/T131024#2153885 (10Huji) [15:44:04] Cyberpower678: 20 is a bit too much. 3 times is more than sufficient. If it still doesn't work, it's not a temporary problem. [15:44:06] Niharika, sure. Alternatively, you can also set the table to filter out every revid=0 [15:45:17] Cyberpower678: Yes, but I think we should keep those stats available to see how often the API doesn't return a valid value. [15:46:48] Does anyone have a python script (or knows of one) that can find a template inside a page's text? [15:48:11] like if we have "... {{tpl1|param1=blah|param2={{tpl2|param=foobar}}|param3=loerm}} ..." [15:48:18] I want the entire thing for tpl1 [15:50:04] Niharika, I added some more details in the output generator. [15:50:16] Cyberpower678: Okay. [15:50:29] So any edit that fails should have an accompanying error message with the article. [15:53:11] Niharika, https://github.com/cyberpower678/Cyberbot_II/blob/master/IABot/API.php#L740 [15:53:51] Cyberpower678: Thanks. [15:56:47] Thy Catafalque failed because of a nobots template [15:57:03] Analyzing Thy Catafalque (25531506)... [15:57:03] Rescued: 1; Tagged dead: 0; Archived: 0; Memory Used: 15.75 MB; Max System Memory Used: 114.75 MB [15:57:04] ERROR: RESTRICTED BY NOBOTS!! [16:35:01] (03Abandoned) 10MarcoAurelio: Adding .lighttpd.conf file [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/276209 (owner: 10MarcoAurelio) [16:35:56] (03Abandoned) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [16:59:42] Niharika, [16:59:43] Analyzing @earth (32543187)... [16:59:43] Rescued: 2; Tagged dead: 0; Archived: 0; Memory Used: 14.25 MB; Max System Memory Used: 37 MB [16:59:44] EDIT ERROR: Received a bad response from the API. [16:59:44] Response: [17:00:50] It would seem the failed edits are a result of nothing being returned by the API [17:03:12] Cyberpower678: Ah, okay. [19:48:32] 10Quarry: Quarry stuck on "queued" state - https://phabricator.wikimedia.org/T131024#2154047 (10yuvipanda) I restarted the workers, which has fixed this for now. Long term, I should migrate quarry to tools / kubernetes for added stability. Apologies for the disruption, and thank you for filing the bug! Some q... [19:48:37] 10Quarry: Quarry stuck on "queued" state - https://phabricator.wikimedia.org/T131024#2154048 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [21:33:38] Niharika, ping [23:24:33] anomie: Are you currently here? [23:27:22] hi! I cannot login to rowiki with tools/taxonbot and get back wrongtoken. But why? Are bots disabled in rowiki? [23:28:25] general question: Whats the difference between badtokken and wrongtoken? rowiki throws Wrongtoken, but I only know badtoken