[00:37:57] PROBLEM - Host tools-docker-builder-01 is DOWN: CRITICAL - Host Unreachable (10.68.19.180) [01:03:39] 06Labs, 10Tool-Labs, 10DBA, 13Patch-For-Review: Tool Labs queries die - https://phabricator.wikimedia.org/T127266#2730527 (10Dispenser) Queries are being killed after 2 hours since June, leaving data for tools stale. [02:14:49] !log analytics restarted druid101 because it was in some kind of kernel lockup [02:14:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Analytics/SAL, dummy [06:55:18] PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:55:59] PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:31:00] RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [07:31:21] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2730717 (10jcrespo) I suggest we should avoid providing wildcard grants. Those have created issues in the past, and we want to move away from them. When g... [07:35:18] RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [11:26:10] 06Labs, 06Operations: Kill the labtest $realm - https://phabricator.wikimedia.org/T148717#2730983 (10faidon) [12:33:06] 06Labs, 06Operations, 10Wikimedia-Video, 07Need-volunteer: Upload the Wikimania 2014 videos to Commons - https://phabricator.wikimedia.org/T106038#2731269 (10Matanya) 05Open>03Resolved This was done by volunteers by uploading lower quality to commons. [12:37:12] hello guys i got a simple question [12:37:26] is there anybody here? :-) [12:55:33] hi [13:41:08] andrewbogott: alive ? :) [13:41:43] partly! What's up? [13:42:21] thanks for the quota increase, though, as i am a ranter, it is not enough :) [13:42:46] I need 4 CPU with 16 GB ram [13:43:06] That's what I did, or at least what I tried to do... [13:43:10] do you have a link to the ticket? [13:43:48] since there is no such image type, i tried to go with m1.xlarge, but that is above my quota [13:43:51] andrewbogott: https://phabricator.wikimedia.org/T147013 [13:44:15] 06Labs, 10Tool-Labs, 13Patch-For-Review: Tools: Migrate puppet roles from ldap to the new puppetbackend - https://phabricator.wikimedia.org/T148683#2731404 (10Andrew) btw, note that the Trusty nodes I just made (tools-exec-1411 through 1420) have their roles set via horizon. So you'll want to remove those a... [13:44:24] ah, I see [13:44:33] didn't we make you a custom flavor for video already? [13:44:43] we have [13:44:46] gigantic [13:45:01] I wouldn't mind going for m1.xlarge [13:45:11] that was my original intention [13:45:20] but i saw we had capacity issues [13:45:39] so requested lower capacity [13:46:40] ok [13:46:42] hang on... [13:47:03] ok, you should be able to make an xlarge now [13:47:10] I think that's the easiest way to go forward for the moment. [14:07:45] thanks andrewbogott [14:08:17] hi [14:08:35] i have submitted a task with "jsub -N name -M user@host -m beas -l release=trusty script.sh" [14:08:51] and I would like to run the task monthly with a cron [14:09:11] what is the command for that? [14:18:37] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 486.10 ms [14:21:26] PROBLEM - Puppet run on tools-exec-1420 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [14:26:26] RECOVERY - Puppet run on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [14:35:04] !log maps Sync to labstore1003 complete (T147657) [14:35:06] T147657: Move maps share to labstore1003 - https://phabricator.wikimedia.org/T147657 [14:35:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL, Master [14:36:03] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [14:41:19] (03CR) 10Lokal Profil: Mount SQL files via a volume instead of copying in Docker setup (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [14:44:31] andrewbogott: thanks +1 (I'm late I guess) [14:45:22] Wesalius: see $ man crontab [14:48:24] (03CR) 10Lokal Profil: [C: 032] Mount SQL files via a volume instead of copying in Docker setup [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [14:50:38] RECOVERY - Puppet run on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [14:51:04] PROBLEM - Host tools-exec-1416 is DOWN: CRITICAL - Host Unreachable (10.68.23.120) [14:54:49] (03Merged) 10jenkins-bot: Mount SQL files via a volume instead of copying in Docker setup [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [15:03:42] RECOVERY - Host tools-exec-1416 is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms [15:07:11] !log maps Symlinks for /home and /data/project from labstore1003 mount setup. Puppet enabled everywhere (T147657) [15:07:12] T147657: Move maps share to labstore1003 - https://phabricator.wikimedia.org/T147657 [15:07:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL, Master [15:10:23] PROBLEM - Host tools-exec-1416 is DOWN: CRITICAL - Host Unreachable (10.68.20.107) [15:14:59] I'm trying to rebuild tools-exec-1416 so there will probably be some more of those [15:18:21] zhuyifei1999_: when I run crontab -e, it defaults to some temporary directory, where should I save my cron file to? [15:19:20] zhuyifei1999_: or should I leave it as is? [15:23:09] just leave the filename as is [15:23:19] k [15:23:31] crontab -l gives the right crontab [15:23:37] so I will hope it is fine [15:23:41] thanks :-) [15:24:11] np :) [15:47:33] (03CR) 10Lokal Profil: [C: 031] "Looks like everything is working but my annoying "No module named pywikibot" issue in tox means I cant try it properly." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314172 (owner: 10Jean-Frédéric) [15:57:23] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.65 ms [16:05:03] addshore: I took a look at the grafana json datasource, please add me to code reviews though, irc isn't always reliable :)) [16:13:30] PROBLEM - Host tools-secgroup-test-103 is DOWN: PING CRITICAL - Packet loss = 100% [16:16:48] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:17:58] (03PS3) 10Lokal Profil: Enable YAML linting via tox using yamllint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309844 (owner: 10Jean-Frédéric) [16:28:09] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.68 ms [16:43:11] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [16:51:47] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [16:52:34] yuvipanda: bastion-03 is time to time very slow this hour [16:55:19] !log tools killed bzip2 taking 100% CPU on tools-bastion-03 [16:55:23] doctaxon: that should be better [16:55:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:55:39] thank you [16:57:30] !log git testing –XX:+UseG1GC on jvm on gerrit-test (gerrit) and if successful will hopefulyl roll out to production [16:57:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL, Master [17:07:28] !log tools.replacer kill bzip2 on tools-bastion, was causing instance to freeze up again [17:09:55] doctaxon: https://wikitech.wikimedia.org/wiki/User_talk:Benjavalero#bzip2_on_tools-login.wmflabs.org I left a message on their user page [17:11:21] okay [17:13:58] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2732127 (10AlexMonk-WMF) >>! In T148560#2730717, @jcrespo wrote: > Right now we maintain a blacklist on realm.pp. We should transform that into a white li... [17:14:59] !log git gerrit-test using just -XX:+UseG1GC failed but doing -Xmx50m -Xms50m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 worked. [17:15:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL, Master [17:17:48] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:21:43] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2732172 (10AlexMonk-WMF) I suppose we go from one list that needs updating when private wikis are created to one that needs updating when non-private (the... [17:40:19] hello, I hope you can help me [17:40:35] I am trying to find an uncompressed XML dump in dev.tools.wmflabs.org [17:40:46] but I can only find bzip ones [17:42:49] benjavalero, where are you looking? [17:43:45] in /public/dumps [17:44:52] benjavalero, /public/dumps/public/metawiki/20161001/metawiki-20161001-abstract.xml exists [17:48:29] yes, I know, but I am interested in "pages-articles", containing the content of the articles [17:49:50] in my local machine, I always download the bzip and uncompress it [17:50:11] but in dev.tools it seems the uncompression uses too much CPU :-( [17:51:03] benjavalero: sorry, I was wrong, there are no uncompressed ones [17:51:11] I think you'll have to process them when compressed [17:51:17] I don't think uncompressing such massive files on tools will work [17:51:27] I have done it in the past [17:51:35] but in dev.tools [17:51:48] well, rather, in the past when those happened it caused massive issues ont eh NFS server [17:51:55] you're accessing an NFS share [17:51:56] and so we've put limits in place that prevent that from happening [17:51:59] it's not actually hosted on dev.tools [17:52:01] since when issues happened on the NFS server [17:52:04] it affected all instances on labs [17:52:08] ok sorry then [17:52:12] my bad [17:52:16] benjavalero: I'd reccomend using a library such as http://pythonhosted.org/mwxml/ to read through the compressed XML [17:52:21] benjavalero: no worries :) [17:52:41] I didn't know that was possible, I will check it [17:52:45] thx!! [17:53:30] that halfak guy makes all the cool big data tools :) [17:54:29] \o/ [17:57:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:16:17] PROBLEM - Host tools-exec-cyberbot is DOWN: CRITICAL - Host Unreachable (10.68.16.39) [19:12:47] yuvipanda: high replication lags now [19:13:35] doctaxon: there seems to be no replag http://tools.wmflabs.org/replag/ [19:14:15] i get replag: 5s for a long time now [19:14:19] bastion-03 [19:15:03] ? [19:15:06] I don't understand [19:17:00] yuvipanda: as example - tools.taxonbot - see aan.out [19:17:35] doctaxon: sorry, can you file a bug with specifics? From what I can see, there's no replag in the labsdbs, and I do not have the time to dig through specific tools' code [19:17:59] it's an .out file, not a code [19:18:48] still, you're much more likely to be able to get help if you write a ticket pointing out the problem. [19:19:48] since the obvious thing to check (tools.wmflabs.org/replag) tells me there's no issues [19:22:00] yuvipanda: it's running now, don't know, what it was [19:22:14] ok :) [19:24:00] 06Labs, 10Tool-Labs, 13Patch-For-Review: Tools: Migrate puppet roles from ldap to the new puppetbackend - https://phabricator.wikimedia.org/T148683#2732561 (10Andrew) [19:45:33] !log git doing -XX:+UseConcMarkSweepGC works in gerrit-test3, just gerrit-test is playing up, calling this a success. [19:45:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL, Master [19:47:16] yuvipanda: hi, can I ask something about PAWS? [19:47:19] yuvipanda: Having troubles seeing results https://commons.wikimedia.org/wiki/Commons:Bots/Work_requests#Categorization_of_Animated_GIFs query: https://quarry.wmflabs.org/query/13531 [19:53:57] 06Labs, 10Tool-Labs, 10MediaWiki-extensions-WikimediaMaintenance: Make maintance script for sending annual survey emails - https://phabricator.wikimedia.org/T148783#2732629 (10bd808) [20:18:06] 06Labs, 06Services (watching): Request increased quota for labs project - https://phabricator.wikimedia.org/T148788#2732756 (10Pchelolo) [20:18:17] 06Labs, 06Services (watching): Request increased quota for services labs project - https://phabricator.wikimedia.org/T148788#2732773 (10Pchelolo) [20:19:00] 06Labs, 06Services (watching): Request increased quota for services labs project - https://phabricator.wikimedia.org/T148788#2732756 (10yuvipanda) I think we should fix that bug rather than increase quota. [20:19:25] 06Labs, 06Services (watching): Request increased quota for services labs project - https://phabricator.wikimedia.org/T148788#2732798 (10yuvipanda) (I'll try to look at it later this week / early next week). Will also happily merge / CR patches before that tho :) [20:19:40] Dispenser: I think the resultset is maybe just too big [20:19:51] I'm figuring out something with PAWS now, I'll take a look at it after [20:20:03] Wesalius: in general, you should just ask the question :) other people might be able to help too [20:20:10] k [20:20:41] but it connects to what we were discussing yesterday, thats why I am targeting you ;-) [20:21:03] indeed, which is why we should've had that conversation here in public :) [20:21:13] PAWS is also running into some issues unrelated to pywikibot now, so I'm trying to fix those [20:21:27] I cloned pwb to my PAWS directory since I am not using the PAWS shared one. When I cd to the pwb dir and run a pwb.py login it does not take the user-config from the current directory but from the home (if there is none it defaults to test:wiki). Is there a way to make pwb use the user-config from the directory the pwb.py resides at? [20:21:30] Wesalius: in general, I also know mostly nothing about pywikibot itself, so pywikibot questions might be better answered in #pywikibot (or by other people here) [20:21:46] 06Labs, 06Services (watching): Request increased quota for services labs project - https://phabricator.wikimedia.org/T148788#2732829 (10Pchelolo) @yuvipanda Thank you. If it's easier to fix the bug - it's great. I've created this request just to have a time bound on either resolving or working around this prob... [20:21:51] Wesalius: see, that's very much a pywikibot quesiton and I've no idea what the answer is [20:21:55] :) [20:22:03] so asking in #pywikibot is going to get you better help I think [20:22:22] it is not, default behaviour of pwb is it looks at user-config in the dir it is ran from [20:22:28] on paws it acts differently [20:22:32] it is a paws issue [20:23:02] the PYWIKIBOT2_USER_DIR environment variable [20:23:55] ? [20:25:41] sorry, PYWIKIBOT2_DIR [20:25:41] https://www.mediawiki.org/wiki/Manual:Pywikibot/user-config.py [20:27:36] The content of the environment variable PYWIKIBOT2_DIR (core) [20:27:41] how do I change the content? [20:28:19] on my paws? [20:28:54] export PYWIKIBOT2_DIR=... [20:28:57] or just use -dir:... [20:29:10] e.g. -dir:. to use the current dir [20:35:31] thanks :-) [20:35:48] now... can I connect to my paws over SCP? [20:35:59] nope [20:36:16] there's an 'upload' button in the main interface, but doesn't work for very large files [20:36:21] so for uploading I am limited to the browser UI? [20:36:24] yes [20:36:27] yeah [20:36:30] we'll hopefully eventually fix this [20:36:40] (also PAWS is having some stability issues right now, and I'm investigating) [20:37:28] today morning I tried to upload some ~2 Mb files, didnt go through [20:37:37] like first 4 did and then it failed [20:38:20] again, PAWS is having some stability issues right now, and I'm working on fixing those [20:38:24] ok [20:38:29] so hopefully it'll be better in a while [20:38:37] np, I was just curious [20:38:41] but yes, uploading is mostly broken for non-trivial files for a while [20:47:38] ok, thanks so far, bye [21:06:17] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Research-and-Data, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2732904 (10bd808) Emails are going out! [21:28:26] (03Draft1) 10Zppix: Edit Project Config [labs/tools/ZppixBot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/317029 [21:28:33] (03CR) 10Zppix: [C: 032 V: 032] Edit Project Config [labs/tools/ZppixBot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/317029 (owner: 10Zppix) [21:38:49] 06Labs, 10Tool-Labs: Send "are you there?" email to tool labs members every 3 months to revalidate email address - https://phabricator.wikimedia.org/T148792#2732932 (10bd808) [21:44:27] 06Labs, 10Tool-Labs: Send "are you there?" email to tool labs members every 3 months to revalidate email address - https://phabricator.wikimedia.org/T148792#2732932 (10Platonides) Looks like //yet another annoying thing// to me… [21:44:46] sorry :P [21:47:26] (03PS1) 10Legoktm: Add .gitreview [labs/tools/suggestor] - 10https://gerrit.wikimedia.org/r/317048 [21:47:28] (03PS1) 10Legoktm: Add a license [labs/tools/suggestor] - 10https://gerrit.wikimedia.org/r/317049 [21:47:45] (03CR) 10Legoktm: [C: 032] Add .gitreview [labs/tools/suggestor] - 10https://gerrit.wikimedia.org/r/317048 (owner: 10Legoktm) [21:47:51] (03CR) 10Legoktm: [C: 032 V: 032] Add a license [labs/tools/suggestor] - 10https://gerrit.wikimedia.org/r/317049 (owner: 10Legoktm) [21:47:53] (03CR) 10Legoktm: [V: 032] Add .gitreview [labs/tools/suggestor] - 10https://gerrit.wikimedia.org/r/317048 (owner: 10Legoktm) [22:09:22] !log project-proxy failover novaproxy to novaproxy-02, upgrade kernel to 4.4, reboot [22:09:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL, Master [23:03:28] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Talfers was created, changed by Talfers link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Talfers edit summary: Created page with "{{Tools Access Request |Justification=My MediaWiki fishing resource website. |Completed=false |User Name=Talfers }}" [23:11:44] 06Labs, 10Tool-Labs: Send "are you there?" email to tool labs members every 3 months to revalidate email address - https://phabricator.wikimedia.org/T148792#2733394 (10bd808) >>! In T148792#2732948, @Platonides wrote: > Looks like //yet another annoying thing// to me… Do you have other suggestions for keeping...