[00:11:36] (03PS5) 10Paladox: Replacing swig with swig-templates [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320294 [00:40:45] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [02:05:02] andrewbogott: i think it is still used but probably can be setup on a new instance [02:05:09] need to check with hoo [02:11:53] aude: I did my best to save it but you might want to check and make sure things are working and puppet is running [02:14:49] andrewbogott: thanks [02:14:57] i think it's somewhat easily replacable [06:37:14] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:11:46] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [09:43:26] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [09:54:24] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [10:16:40] 06Labs, 10Tool-Labs, 15User-Addshore, 15User-bd808: Possible use of tools-lab-elasticsearch cluster - https://phabricator.wikimedia.org/T149709#2778963 (10Tarrow) Great! Can you add a user for 'tools.wikifactmine-pipeline'? Thanks. [10:27:45] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms [10:33:18] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 3.15 ms [10:35:32] PROBLEM - Host tools-secgroup-test-103 is DOWN: PING CRITICAL - Packet loss = 100% [10:38:15] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [11:13:19] (03PS1) 10Jean-Frédéric: Set-up Raven client for error reporting via Sentry [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/320363 (https://phabricator.wikimedia.org/T137882) [14:12:53] Having some instability with ores.wmflabs.org and I'm afk. anything interesting going on? [14:13:16] halfak|Mobile: network issues afaics [14:13:34] Thanks. [14:52:24] hey, can anyone see the conversation between Platonides and I? It's gone already for me [14:55:02] Also, current status on my bot task per qstat is rR [14:56:42] (I've stopped it now as it wasn't working anyways. Need to bugfix per above) [14:57:24] DatGuy: this channel is publicy logged, see the url in the topic to recover it :) [14:57:36] I've just joined so I can't see it [14:58:17] cheers, I'm checking it out [15:05:20] http://pastebin.com/rYYk1KC5 mafk [15:11:44] what should I do? [15:14:00] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2779859 (10yuvipanda) I wonder if it'll be better to do this next quarter. We've already done a few bits of pretty disruptive maintenance, and have on... [15:36:07] Any ideas? [15:39:05] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2779893 (10jcrespo) > I wonder if it'll be better to do this next quarter. I am ok with next quarter- let's set a time. I have workarounded the 5.5 s... [15:47:08] are user rights changes logged in the `logging` table, and if so, what log_type are they? [15:51:51] yes, rights [15:55:56] Krenair: *sigh*, thank you [16:03:01] (03CR) 10Lokal Profil: [C: 032] Add Wikidata to the database statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/319986 (owner: 10Jean-Frédéric) [16:05:17] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2779979 (10yuvipanda) Ok. Early January? [16:08:52] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2779984 (10jcrespo) January ok, but after the 15th. [16:09:52] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:13:34] (03Merged) 10jenkins-bot: Add Wikidata to the database statistics [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/319986 (owner: 10Jean-Frédéric) [16:17:38] !log tools.heritage Deploy latest from Git master: a70289c (T149258), c7eb06a, b5aeb29, 828e309 [16:17:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [16:17:41] T149258: ErfgoedBot categorisation task fails with TypeError - https://phabricator.wikimedia.org/T149258 [16:27:30] (03Draft1) 10Paladox: Reply back permission denied if you do not have access to the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320419 [16:27:33] (03Draft2) 10Paladox: Reply back permission denied if you do not have access to the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320419 [16:28:09] (03CR) 10jenkins-bot: [V: 04-1] Reply back permission denied if you do not have access to the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320419 (owner: 10Paladox) [16:29:21] (03PS3) 10Paladox: Reply back permission denied if you do not have access to the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320419 [16:49:55] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [17:09:21] 06Labs, 10Beta-Cluster-Infrastructure, 10Wikimedia-General-or-Unknown: rename -labs.php to -beta.php - https://phabricator.wikimedia.org/T150268#2780096 (10Reedy) [17:10:31] yuvipanda: ^ I'll do that tonight [17:11:06] Reedy: \o/ <3 [19:45:36] (03CR) 10Gergő Tisza: "If you have version-specific use cases, you are welcome to add them to T128911." (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/320363 (https://phabricator.wikimedia.org/T137882) (owner: 10Jean-Frédéric) [20:25:40] hi, can someone update python-twisted in ToolLabs? the last version is 16.5, Tool Labs has the 13.2, I need a feature that is new in 14 [20:27:13] danilo: you should use a virtualenv and then you can use whatever version you want [20:28:15] ok [20:30:26] I also need to use a feature in tools.labsdb that is new in MariaDB 10.0.1, but tools.labsdb has the version 5.5.39 [20:31:00] danilo: yes, that will probably be able to change only in January of next year when we upgrade it [20:31:17] ok [20:56:50] !log snuggle stopped apache2 service, started database backup (migration) [20:56:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Snuggle/SAL [21:06:40] !log snuggle running mongorestore on snuggle-enwiki-01 [21:06:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Snuggle/SAL [21:27:59] (03Draft1) 10Paladox: Make irc commands based on the irc bots nick [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320457 [21:28:02] (03Draft2) 10Paladox: Make irc commands based on the irc bots nick [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320457 [21:28:49] (03CR) 10Zppix: [C: 032] Make irc commands based on the irc bots nick [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320457 (owner: 10Paladox) [21:28:59] (03CR) 10Zppix: [C: 032] Reply back permission denied if you do not have access to the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320419 (owner: 10Paladox) [21:29:10] (03CR) 10Zppix: [C: 032] grrrit-wm: Improve logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:31:06] (03CR) 10MarkTraceur: [C: 04-1] "Ugh, no" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:31:50] (03CR) 10Paladox: [C: 04-2] grrrit-wm: Improve logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:32:06] (03Merged) 10jenkins-bot: Make irc commands based on the irc bots nick [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320457 (owner: 10Paladox) [21:32:09] (03Merged) 10jenkins-bot: Reply back permission denied if you do not have access to the irc commands [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320419 (owner: 10Paladox) [21:32:12] (03CR) 10jenkins-bot: [V: 04-1] grrrit-wm: Improve logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:32:27] (03CR) 10MarkTraceur: "This isn't reliable, because again, you're checking if the connection has been established, but if it hasn't, you just don't log the recon" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:32:41] (03CR) 10Zppix: grrrit-wm: Improve logging [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:33:35] (03CR) 10Paladox: "> This isn't reliable, because again, you're checking if the" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:34:45] (03CR) 10Paladox: "@MarkTraceur" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:36:30] (03CR) 10MarkTraceur: "OK," [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320214 (owner: 10Paladox) [21:44:05] (03PS1) 10Paladox: Update ssh2 to 0.5.3 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320480 [21:45:11] (03CR) 10Paladox: [C: 032] Update ssh2 to 0.5.3 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320480 (owner: 10Paladox) [21:47:00] paladox: Why are you self-merging patches on a bot we use for production-level work [21:47:29] marktraceur i was just about to +2 it myself [21:47:43] I can stop if you doint like what i am doing [21:47:53] paladox: I'm just trying to make sure grrrit-wm doesn't have problems [21:48:05] (03Merged) 10jenkins-bot: Update ssh2 to 0.5.3 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320480 (owner: 10Paladox) [21:48:10] Oh, yep, i look through the changelog [21:48:10] paladox: And I hope you checked that the update didn't have any breaking changes, but I don't know, because you did it yourself [21:48:31] https://github.com/mscdex/ssh2/commits/master [21:48:41] paladox: Anyway, in general, production-level tools are not meant to be self-reviewed [21:48:44] ^^ no breaking change [21:48:48] Ok [21:48:49] sorry [21:49:10] And while there might be some debate about the production status of grrrit-wm, and I welcome people to chime in here, I personally rely on that bot to inform me about patches\ [21:51:37] marktraceur so do i, which is why we havent restarted the prod version even though there was now 3-4 changes merged] [21:52:31] Zppix: As long as those patches go through an appropriate review process, I'm fine with applying them to the bot [21:53:01] Zppix: I'm just saying that there have been multiple issues with self-merging and questionable coding practices (the latest race condition being only one example) that really worry me [21:54:50] marktraceur trust me if i saw a huge issue that would prevent the bot from running or semi-functioning i would pull the patch outta the prod bot [21:55:38] Zppix: It's not a question of trusting you, I'm sure you're fine, but I don't want anyone self-merging in a repo we rely on [21:55:56] marktraceur we test all our changes with the test bot we have [21:56:02] Zppix: And even if the bot is running, if it's slow or unreliable or we can't debug what's wrong (because the logging is inaccurate), that's a real issue [21:56:19] paladox: It's not a matter of testing, it's a matter of review [21:56:30] Oh ok [21:56:50] paladox: This is not a practical matter entirely, though I think I've pretty accurately located several issues in merged code that should be fixed [21:57:08] paladox: We have a social contract in Wikimedia technical projects that we don't just go around merging our own changes willy-nilly [21:57:32] paladox: And as far as I can tell, that's exactly what has happened, multiple times, over the past two weeks or so in the grrrit repository. [21:57:44] Yes sorry [21:57:53] but as you seen i have been doing less of self merging [21:58:00] marktraceur to be fair the changes for most part all that major only 1-2 have been [21:58:22] Zppix: That's a good point, but even the major change was, at least in part, a self-merge [21:58:49] Zppix: And even small changes can have profound effects on the operation of the bot, and the surrounding infrastructure [21:58:53] Which im sorry for, and havent done any majour changes mergers since. [21:59:47] paladox: Let's change "major changes" to "changes", stop merging your own patches, and let's just let code review happen the way it's meant to [22:00:18] Ok [22:00:52] paladox: If you need CR you can ping me or a number of people who have reviewed JS in the past [22:01:00] Same goes for Zppix [22:01:04] Oh thanks, [22:03:41] marktraceur honestly i don't plan on merging anything thats mine unless its in my repo (for my bot) [22:04:04] Zppix: 100% fine as long as the bot isn't vital infrastructure for WM tech [22:04:41] Zppix: Though, not to be pedantic, I recall you working on the original grrrit-wm: restart patch, and I believe you were the one to merge [22:04:58] I don't have it in front of me, so maybe I'm wrong, but that's what I remember [22:06:52] marktraceur my bot is an irc bot lol [22:07:06] marktraceur ah yes i did merge it [22:07:45] but the only "changes" i did was rebasing and i believe a little tiny code changes (i think) [22:08:08] Zppix: I guess early on you were working on the whitelist parts but those never made it into the first patch? [22:08:20] Which is a whole different thing but whatever I'm willing to focus on this now [22:10:20] marktraceur one thing i've learned over the years is if you think it might break it dont do or ask billions of questions beforehand [22:10:48] Yeah [22:14:07] mhashemi: hey! leila pinged me and said you might need some help? [22:14:35] yuvipanda: thanks for coming in. can you check your email? mhashemi has provided some details. [22:15:04] For a crontab line, how do I make sure that my venv is used? [22:15:19] this is my current line: [22:15:20] 0 20 * * * jsub -once -N mpcdailydigest -mem 7g /data/project/mediaplaycounts/www/python/venv/bin/python3 /data/project/mediaplaycounts/www/python/src/mediaplaycounts/DailyIngest.py [22:16:02] despite this being the entry, the script errored out saying that a given module didn't exist, likely because it was not using the virtualenv [22:19:54] leila: I don't really understand that email [22:20:14] * yuvipanda re-reads [22:21:02] yuvipanda: just pinged mhashemi. [22:21:38] did bastion just die?! [22:22:37] yuvipanda why you think that? [22:23:04] yuvipanda: while we're waiting for mhashemi: what I understood is that the query he has put in the email works in quary, but doesn't work in Labs. [22:23:32] and if that doesn't run, yuvipanda, mhashemi cannot get all the information he needs to get for each img_name [22:23:49] leila: I just ran that query from the commandline and it works for me [22:24:01] I'll just wait for mhashemi :) [22:36:52] (03PS1) 10Paladox: Remove hard coded grrrit-wm: irc command [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 [22:37:13] marktraceur hi, could you review ^^ please? [22:38:25] harej: does that work if you run it from the commandline? [22:38:37] yuvipanda: yes [22:39:35] (03CR) 10MarkTraceur: Remove hard coded grrrit-wm: irc command (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:40:59] harej: hmm [22:41:13] yuvipanda: it works after i do `source` of course [22:41:26] hmm interesting. [22:41:32] that shouldn't be entirely needed [22:41:33] but i thought the point of doing that whole rigamarole of invoking that specific python executable was that it would use that environment [22:41:39] but maybe cron has weird interactions with it? [22:41:40] yeah [22:41:59] project is mediaplaycounts if you want to take a look [22:42:27] (03CR) 10Paladox: Remove hard coded grrrit-wm: irc command (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:42:31] I'm helping the wlm people, will look after [22:42:45] (03CR) 10Zppix: "@MarkTraceur i can see regex causing issues possibly if something were to be added into the code incorrectly thus possibly giving access " [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:43:22] (03CR) 10Paladox: "the whitelist should prevent this." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:43:39] paladox i know that but you can never be to safe [22:44:04] harej: I pinged madhuvishy, hopefully she can take a look [22:44:12] Zppix well someone will have to know that users password to gain there hostname [22:44:18] IE mediawiki/paladox for example [22:44:55] (03CR) 10MarkTraceur: "@Zppix, exactly how would using a regex to find the bot's nick + ": restart" have any bearing on security?" (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:45:08] harej: use the python binary created by the virtualenv [22:45:18] I do? [22:45:38] Is the binary not /data/project/mediaplaycounts/www/python/venv/bin/python3 ? [22:45:57] probably [22:46:08] it doesn't work? [22:46:40] (03CR) 10Zppix: "i mis-commented what i meant was it could cause some security issues via say making another auto response/bot that *could* be whitelisted " [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:46:43] i'm looking too [22:47:22] you can always call a script that sources the virtualenv and runs whatever is needed [22:47:25] that must work [22:47:39] (03PS2) 10Paladox: Remove hard coded grrrit-wm: irc command [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 [22:47:45] (03CR) 10MarkTraceur: [C: 04-1] "Literally none of that is how regexes work, but my inline comment seems to be the real issue, so if paladox could reply that would be grea" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:48:03] Platonides: basically a two line shell script, source whatever/activate ; python3 file.py? [22:48:05] (03CR) 10Paladox: Remove hard coded grrrit-wm: irc command (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:48:11] yes [22:48:23] I'd puyt a shebang, too [22:48:37] paladox: I admit that I don't have the capacity to test the bot, so could you fire up your test version and point me to the channel it's in? [22:48:53] Ok [22:48:57] marktraceur i may be completely wrong honestly my regex is a little rusty when it comes to javascript dev [22:49:04] marktraceur it's in #wikimedia-bot-gerrit [22:49:35] Platonides: i could well resort to that, but i shouldn't have to? I've done Python in virtualenvs in crontab before. [22:49:57] marktraceur i sent you an invite [22:50:02] harej: hmmm source shouldn't be necessary [22:50:06] Zppix: A regex tests a string. So e.g. new Regex( '/^' + ircClient.nick + ': restart/' ); would be able to detect whether the string contains the relevant command. [22:50:08] http://stackoverflow.com/questions/3287038/cron-and-virtualenv [22:50:18] Zppix: It has nothing to do with the whitelist or whatever an "auto-response" is. [22:50:32] marktraceur disregard my comments then i mis intererpted [22:51:28] harej: I thought the python command there would be able to [22:51:36] as did I :( [22:51:40] and that's indeed wha tthe accepted answer states [22:53:38] maybe the problem isn't really with the virtualenv? [22:53:43] i think may be it needs to be on the path [22:54:50] cd to working directory and then python .. [22:54:51] harej: madhuvishy http://stackoverflow.com/questions/3287038/cron-and-virtualenv [22:54:54] yeah [22:55:10] yeah that's the link i shared ^ [22:55:34] cd to the executable? or to where the file is? [22:55:51] What would the new command be... [22:57:58] harej: what are you attempting to run? [22:58:13] harej: to where your venv is? [22:58:29] 0 20 * * * jsub -once -N mpcdailydigest -mem 7g cd /data/project/mediaplaycounts/www/python/src/mediaplaycounts && /data/project/mediaplaycounts/www/python/venv/bin/python3 /data/project/mediaplaycounts/www/python/src/mediaplaycounts/DailyIngest.py [22:58:42] hmm [22:58:52] the cd is done by jsub [22:58:53] (03CR) 10MarkTraceur: [C: 032] "Thanks for running through CR with me so quickly, guys." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [22:59:04] and python executed by cron [22:59:15] probably not what you intended :P [22:59:18] Thank you makrtraceur ^^ and your welcome :) [22:59:31] uhh i might have typed wrong cd first and then jsub [23:00:22] still, the virtualenv seems to activate right [23:00:45] Platonides: a script to analyze a very large log file and input the outcome of the analysis into a database, run daily [23:00:46] this works: /data/project/mediaplaycounts/www/python/venv/bin/python -c 'import arrow' [23:00:52] this fails: python -c 'import arrow' [23:01:06] In crontab? [23:01:23] in the shell [23:03:08] * Platonides notices a VIRTUALENVWRAPPER_SCRIPT=/usr/share/virtualenvwrapper/virtualenvwrapper.sh in the default environment [23:08:44] (03CR) 10jenkins-bot: [V: 04-1] Remove hard coded grrrit-wm: irc command [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [23:11:12] (03PS3) 10Paladox: Remove hard coded grrrit-wm: irc command [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 [23:11:36] (03CR) 10Paladox: "Fixed syntax, was just failing jshint but actually worked with the test bot." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [23:11:53] (03CR) 10Paladox: [C: 032] "Re +2 per MarkTraceur" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [23:19:31] (03Merged) 10jenkins-bot: Remove hard coded grrrit-wm: irc command [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320532 (owner: 10Paladox) [23:22:11] 10Labs-Kubernetes, 06Community-Tech, 06Wikisource: Make Google OCR API on Tool Labs work under Kubernetes - https://phabricator.wikimedia.org/T146311#2781509 (10DannyH) [23:24:56] 10Tool-Labs-tools-Pageviews, 06Community-Tech: Data since 16 October not showing for some users - https://phabricator.wikimedia.org/T149058#2781513 (10DannyH) [23:44:43] (03PS1) 10Zppix: Adding help command to grrrit-wm [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/320541