[02:54:40] !log tools.wikibugs RIP pywikibugs [02:54:44] Logged the message, Master [03:15:59] (03PS1) 10Legoktm: Add .gitreview [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175393 [03:16:10] (03CR) 10Legoktm: [C: 032 V: 032] Add .gitreview [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175393 (owner: 10Legoktm) [03:16:30] wat [03:16:34] I don't have submit permissions? [03:17:19] there. [03:21:31] (03PS1) 10Legoktm: fix flake8 errors [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175395 [04:41:18] 3Wikimedia-Labs-tools: become should complain more clearly if the user is not (in this session) a member of the tool - https://phabricator.wikimedia.org/T70156#777412 (10scfc) [05:17:46] 3Wikimedia-Labs-tools: Package and delete warn-screen - https://phabricator.wikimedia.org/T62924#777495 (10scfc) 5Open>3Invalid ```scfc@tools-login:~$ cat /usr/local/sbin/warn-screen cat: /usr/local/sbin/warn-screen: Datei oder Verzeichnis nicht gefunden scfc@tools-login:~$``` [06:48:43] (03CR) 10Legoktm: [C: 032 V: 032] fix flake8 errors [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175395 (owner: 10Legoktm) [09:37:27] there is a problem with the wikimedia dumps on labs again. what tag on phabricator should i use then? [09:37:41] *which tag [10:34:40] 3Wikimedia-Labs-tools: Webservice being shut down abruptly - https://phabricator.wikimedia.org/T75502#777819 (10zhaofengli) [10:36:00] 3Wikimedia-Labs-General: Some files of the dewiki-20141111 dump missing - https://phabricator.wikimedia.org/T75721#777822 (10Giftpflanze) [10:46:48] 3Tool-Labs-tools-Other: Magnus Treeviews not processing - https://phabricator.wikimedia.org/T56009#777853 (10Magnus) 5Open>3Resolved Should be working. Your example category "Archives of American Art task force articles" does contain talk pages, so don't forget to check the "Talk" checkbox for the category. [11:21:22] Coren: who is responsible again for the wikimedia dumps? i want to add them to the relevant bugs [11:59:36] 3Wikimedia-Labs-deployment-prep-(beta): Need a way to simulate replication lag to test replag issues - https://phabricator.wikimedia.org/T40945#778030 (10hashar) [12:00:45] annika_: should be Ariel Glenn "apergos" [12:30:53] 3Wikibugs, Tool-Labs-tools-wikibugs-IRC-bot, Wikimedia-wikibugs-IRC-bot: Rename phawkes to wikibugs - https://phabricator.wikimedia.org/T75690#778103 (10Aklapper) After this there are now #Wikimedia-wikibugs-IRC-bot and #Tool-labs-tools-wikibugs-IRC-bot and #Wikibugs projects in Phab. Any comment on that by main... [12:46:24] 3Wikimedia-Labs-General: dewiki-20140918-pages-meta-history{3,4}.bz2 dumps are missing - https://phabricator.wikimedia.org/T73967#778132 (10Giftpflanze) 5Open>3Resolved [12:59:28] 3Wikimedia-Labs-deployment-prep-(beta): Safari sets forceHTTPS=deleted incorrectly, causing login failure on Beta Cluster - https://phabricator.wikimedia.org/T72145#778187 (10yuvipanda) Is this still 'unbreak now'? [13:00:13] 3Wikimedia-Labs-deployment-prep-(beta): Rename labswiki to deploymentwiki - https://phabricator.wikimedia.org/T72108#778199 (10yuvipanda) So the patch got merged. Was a manual rename performed? [13:00:54] 3Wikimedia-Labs-Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#778204 (10yuvipanda) [13:00:55] 3Wikimedia-Labs-deployment-prep-(beta): deployment-mediawiki02 labs instance only has 2GB of disk - https://phabricator.wikimedia.org/T71976#778201 (10yuvipanda) [13:02:10] 3Wikimedia-Labs-deployment-prep-(beta): Setup monitoring for Beta cluster - https://phabricator.wikimedia.org/T53497#778209 (10yuvipanda) [13:02:11] 3Wikimedia-Labs-deployment-prep-(beta): Yell loudly of failed puppet runs on Beta Cluster instances - https://phabricator.wikimedia.org/T69333#778206 (10yuvipanda) [13:04:16] 3Wikimedia-Labs-deployment-prep-(beta): Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#778211 (10yuvipanda) @greg @Nik: We have ES metrics in graphite also now (graphite.wmflabs.org) so might make it easier to debug? [13:05:12] 3Wikimedia-Labs-deployment-prep-(beta): obtain access to deployment-bastion.eqiad.wmflabs - https://phabricator.wikimedia.org/T71435#778216 (10yuvipanda) 5Open>3Resolved a:3yuvipanda I am pretty sure he has access now :) [13:06:54] 3Wikimedia-Labs-deployment-prep-(beta): monitor that application servers are responding - https://phabricator.wikimedia.org/T54867#778231 (10yuvipanda) a:3yuvipanda Hmm, so the 'ideal' way is for shinken to hit port 80 on those instances and check if they are serving content properly. This is complicated by fi... [13:22:53] !password [13:22:53] gfgjoagaewhgAW#YAU_#YU*U^*^%Q#Tqyhe [13:31:45] Lol [13:31:47] !password [13:31:47] gfgjoagaewhgAW#YAU_#YU*U^*^%Q#Tqyhe [13:31:55] lol it's actually the same [13:32:07] Yeah [13:33:28] andrewbogott_afk: hmm, you remember the ldap tricks we had to do for mwdeploy long long time ago? [13:38:27] 3Wikimedia-Labs-Infrastructure: When uploading files to web, in network activity is twice of out - https://phabricator.wikimedia.org/T45060#778402 (10faidon) [13:39:04] 3Wikimedia-Labs-Infrastructure: Synchronize mediawiki groups with LDAP have all projects require shell group - https://phabricator.wikimedia.org/T46167#778411 (10faidon) [13:39:48] 3Wikimedia-Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#778413 (10faidon) [13:42:03] 3Wikimedia-Labs-Infrastructure: Add eqiad as a production region - https://phabricator.wikimedia.org/T41789#778430 (10yuvipanda) 5Open>3Resolved a:3yuvipanda 2012 called, wants its unclosed bug back. [13:43:06] 3Wikimedia-Labs-Infrastructure: wikidata (federated?) database not available for many wikis - https://phabricator.wikimedia.org/T75511#778443 (10yuvipanda) [13:43:32] 3Wikimedia-Labs-Infrastructure: mediawiki_singlenode's $wgMainCacheType = CACHE_ACCEL can't be overridden - https://phabricator.wikimedia.org/T74600#778450 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [13:43:37] 3Wikimedia-Labs-Infrastructure: Automatically updated list of all configured domains - https://phabricator.wikimedia.org/T45580#778454 (10faidon) [13:44:08] 3Wikimedia-Labs-Infrastructure: default labs MediaWiki config will generate https links - https://phabricator.wikimedia.org/T58389#778464 (10faidon) [13:46:13] 3Wikimedia-Labs-Infrastructure: Monitor Puppet "stability" - https://phabricator.wikimedia.org/T66964#778478 (10yuvipanda) Ah, hmm. We can use diamond to also calculate the changed events, and then use that as a metric. [13:46:21] 3Wikimedia-Labs-Infrastructure: Create users jenkins and zuul on the NFS server - https://phabricator.wikimedia.org/T66868#778480 (10yuvipanda) @hashar is this still needed? [13:47:26] 3Wikimedia-Labs-Infrastructure: Change I9238a052 broke ircecho on labs - https://phabricator.wikimedia.org/T64407#778490 (10yuvipanda) Action items? [13:48:21] 3Wikimedia-Labs-Infrastructure: accommodate project-specific puppet git branches - https://phabricator.wikimedia.org/T37737#778497 (10yuvipanda) There's the puppetception module that I've written but not used anywhere that lets you use any puppet repo for your labs host rather than the operations/puppet one. Tha... [13:50:00] 3Wikimedia-Labs-Infrastructure: Set up a "file dropbox" or similar for temporary storage of files pending server-side upload - https://phabricator.wikimedia.org/T33828#778505 (10yuvipanda) p:5Low>3Triage [13:50:35] 3Wikimedia-Labs-Infrastructure: Enable development on environments resembling that of Wikimedia Commons - https://phabricator.wikimedia.org/T46837#778508 (10yuvipanda) 5Open>3Resolved a:3yuvipanda I'd say betalabs provides this. [13:58:27] ok, tools-login seems oom [13:58:32] oh [13:58:33] * YuviPanda checks [13:58:51] yup it does [13:59:26] 3Wikimedia-Labs-deployment-prep-(beta): UserMailer silently fails to deliver anything on betacluster - https://phabricator.wikimedia.org/T75736 (10Steinsplitter) 3NEW p:3Volunteer? [14:02:08] !log tools rebooting tools-login, OOM'd [14:02:13] Logged the message, Master [14:03:04] need to setup some monitoring for that as well, I think [14:11:02] 3Wikimedia-Labs-Infrastructure: Create users jenkins and zuul on the NFS server - https://phabricator.wikimedia.org/T66868#778661 (10hashar) 5Open>3declined a:3hashar Since May, I most probably found an acceptable workaround. I can't really remember what was this task for, so closing it. [14:14:31] ok, the puppet failure was just an OOM [14:17:11] 3Wikimedia-Labs-Infrastructure: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#778702 (10hashar) [14:17:57] 3Wikimedia-Labs-deployment-prep-(beta): deployment-mediawiki02 labs instance only has 2GB of disk - https://phabricator.wikimedia.org/T71976#778706 (10yuvipanda) Only way to increase /var is to re-create instances. New instances have a re-sizable /var [14:19:46] 3Wikimedia-Labs-deployment-prep-(beta): HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk - https://phabricator.wikimedia.org/T71976#778712 (10hashar) a:5yuvipanda>3None [14:36:00] YuviPanda: an OOM on virt1000? [14:36:07] andrewbogott: no, tools-login [14:36:12] ah, ok. [14:36:12] I haven't investigated the virt1000 one [14:36:22] virt1000 seems to be working fine now. [14:36:30] yeah [14:36:40] I got about a hundred shinken [14:37:02] That happens in prod sometimes too, I don't quite understand why [14:37:29] shinken alerts, I meant [14:37:35] getting a hundred shinkens sounds fun too [14:37:36] though [14:37:48] andrewbogott: might be a transient restart somewhere. [14:38:37] YuviPanda: I mean, sometimes a million prod hosts complain about a transient puppet failure all at once [14:41:54] andrewbogott: there's some ldap issues causing lots of failures on betalabs. think you'll have time to help / show me? [14:41:57] * YuviPanda hasn't dealt with ldap before [14:42:14] Error: Could not set home on user[mwdeploy]: Execution of '/usr/sbin/usermod -d /home/mwdeploy mwdeploy' returned 6: usermod: user 'mwdeploy' does not exist in /etc/passwd [14:42:16] Error: /Stage[main]/Mediawiki::Users/User[mwdeploy]/home: change from /var/lib/mwdeploy to /home/mwdeploy failed: Could not set home on user[mwdeploy]: Execution of '/usr/sbin/usermod -d /home/mwdeploy mwdeploy' returned 6: usermod: user 'mwdeploy' does not exist in /etc/passwd [14:43:06] Is that happening on precise boxes or just trusty? [14:43:12] andrewbogott: this is a precise box [14:43:34] that's the keyholder stuff [14:43:34] And is it a new bug, or does it result from some puppet change? [14:43:44] hashar already pointed this out on the list [14:44:04] oh? which list? [14:44:06] ops@ [14:44:11] * YuviPanda goes to check [14:44:45] Ah, right, the new deployer auth stuff [14:45:23] paravoid: ah, this is different. That was fixed, I think. [14:45:37] ok [14:46:02] there's a bug for this from a long tim eago [14:46:03] I think [14:46:49] 3Wikimedia-Labs-Infrastructure: wikidata (federated?) database not available for many wikis - https://phabricator.wikimedia.org/T75511#778890 (10Magnus) 5Resolved>3Open I don't care what it's called, the error remains. On shell, I do: sql elwiki_p In SQL shell, I do: MariaDB [elwiki_p]> select * from wiki... [14:47:36] andrewbogott: is a recurrance of https://phabricator.wikimedia.org/T67591 [14:48:05] 3Wikimedia-Labs-Infrastructure: wikidata (federated?) database not available for many wikis - https://phabricator.wikimedia.org/T75511#778910 (10yuvipanda) Apologies, I misread the timestamp from the importer. Will investigate. [14:49:02] maybe someone forgot to keep the local hacks in place [14:49:30] Yeah, I have the feeling this was fixed but in a way that isn't mentioned in that bug [14:51:31] andrewbogott: yeah, it was fixed with local commits hat are gone now [14:51:48] local commits from who? [14:52:37] 3Wikimedia-Labs-deployment-prep-(beta): mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - https://phabricator.wikimedia.org/T67591#778964 (10yuvipanda) This is recurring now since someone has accidentally let the local commits die. [14:52:49] andrewbogott: from bd808|BUFFER I think. [14:53:03] andrewbogott: deployment-prep had two local commits with [LOCAL HACK] or somesuch, that were needed for it to work properly [14:53:24] ok. So maybe we can punt until bd808|BUFFER comments? [14:53:43] yeah [14:53:45] d67f402 HEAD@{34}: pull: [LOCAL HACK] Change MySQL admin user in sql script [14:53:48] and e5b430a HEAD@{35}: pull: [LOCAL HACK] Bug 65591: User['mwdeploy'] shell => /bin/bash [14:53:50] are missing now [14:57:28] andrewbogott: I cherry-picked them, let's see what happens [14:58:03] !log deployment-prep cherry-picked 3e45c538978710113e6e28e9d533bf8d18c159a6 and 9d4614a8a352c78505212fd6e9d2a7be6d2e4927 to deployment-salt puppetmaster, restoring local hacks [14:58:05] Logged the message, Master [15:06:43] Coren: you around? [15:06:52] Betacommand: What's up? [15:08:06] Coren: see PM BEANS [15:16:13] !log deployment-prep modified local hack to account for 47dcefb74dd4faf8afb6880ec554c7e087aa947b [15:16:15] Logged the message, Master [15:30:03] Coren: email sent, and you CCed [15:30:24] Betacommand: kk [15:30:57] I hate phab's search [15:38:26] 3Wikimedia-Labs-tools: webservice truncates files - https://phabricator.wikimedia.org/T74813#779884 (10Magnus) 5Open>3Resolved Since this has mysteriously fixed itself, I'm closing it. [15:40:17] Coren: can you tell me more about the performance changes hoo asked about over the weekend? Or is there a bug open for that? [15:49:50] andrewbogott: Not unless he opened one. The short of it: virt1009 doesn't seem unusually overloaded but when they start runs the /instances/ peg quickly in iowait. [15:50:10] Do you know if that's specific to virt1009? [15:51:02] andrewbogott: They're the only ones having reported issues - doesn't mean it isn't happening elsewhere but nobody noticed and reported it if it does. [15:51:35] So it's not clear if it's an issue with their instances, their project, or the host. [15:52:04] ok, and you don't know if it's disk or network io that was slow? [15:52:21] andrewbogott: The instance is pegged on iowait [15:52:50] andrewbogott: And, according to hoo, this isn't NFS at all but their local filesystem. [15:57:13] YuviPanda: somebody rebased the beta puppet hacks away? [15:58:17] Oh... I think there may be a bug for fixing the ldap homedir for mwdeploy in beta [16:06:12] 3Wikimedia-Labs-deployment-prep-(beta): Rename labswiki to deploymentwiki - https://phabricator.wikimedia.org/T72108#780494 (10bd808) 5Open>3Resolved a:3bd808 We did this a long time ago. [16:06:15] 3Wikimedia-Labs-deployment-prep-(beta): Rename labswiki to deploymentwiki - https://phabricator.wikimedia.org/T72108#780499 (10bd808) [16:10:33] bd808: yup I put them back. Not sure who put them away [16:11:03] bd808: I checked LDAP it is set to /var I think [16:14:03] YuviPanda: The bug I was thinking of is different -- https://phabricator.wikimedia.org/T75206 -- I do remember commenting on the patch where Ori changed the homedir for mwdeploy that we'd need to fix that in beta or labs ldap, but I can't find a bug for it. [16:14:53] Looking [16:15:08] * bd808 is going to be lost in phabricator for a bit until he figures out the right queries to find things [16:25:35] 3Tool-Labs-tools-Other: [tracking] toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#780960 (10Magnus) [16:25:37] 3Tool-Labs-tools-Other: Migrate https://toolserver.org/~magnus/redirector.php - https://phabricator.wikimedia.org/T63193#780957 (10Magnus) 5Open>3Resolved a:3Magnus Done: https://tools.wmflabs.org/magnus-toolserver/redirector.php [16:28:00] bd808: :) [16:28:01] bd808: ldaplist says [16:28:02] homeDirectory: /var/lib/mwdeploy [16:28:04] for mwdeploy [16:35:21] 3Tool-Labs-tools-Other: [tracking] toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#781214 (10Magnus) [16:35:23] 3Tool-Labs-tools-Other: Migrate https://toolserver.org/~magnus/persondata.php - https://phabricator.wikimedia.org/T63189#781211 (10Magnus) [16:52:12] 3Tool-Labs-tools-Other: [tracking] toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#781261 (10Magnus) [16:52:13] 3Tool-Labs-tools-Other: Migrate https://toolserver.org/~magnus/related_places.php - https://phabricator.wikimedia.org/T63177#781258 (10Magnus) 5Open>3Resolved a:3Magnus Done: https://tools.wmflabs.org/magnus-toolserver/related_places.php? [17:35:24] 3Wikibugs, Phabricator, Tool-Labs-tools-wikibugs-IRC-bot, Wikimedia-wikibugs-IRC-bot: Merge wikibugs projects - https://phabricator.wikimedia.org/T75765#781359 (10Legoktm) [17:35:42] 3Wikibugs, Tool-Labs-tools-wikibugs-IRC-bot, Wikimedia-wikibugs-IRC-bot: Rename phawkes to wikibugs - https://phabricator.wikimedia.org/T75690#781367 (10Legoktm) [17:36:29] Hey, does anyone know why this error is happening for me? 2014-11-24 17:34:15: (mod_fastcgi.c.2701) FastCGI-stderr: PHP Warning: shell_exec(): Unable to execute '/usr/bin/python2.7 /data/project/dexbot/pywikipedia-git/maghalesaz_cmd3.py '-faname:برلسک (فیلم ۲۰۱۰ آمریکایی)' '-enname:Burlesque_(2010_American_film)'' in /data/project/dexbot/public_html/c.php on line 14 [17:36:31] PHP Stack trace: [17:36:33] PHP 1. {main}() /data/project/dexbot/public_html/c.php:0 [17:36:34] PHP 2. shell_exec() /data/project/dexbot/public_html/c.php:14 [17:42:01] !log ceph fixed puppet and ldap post-eqiad migration for cephticon3 because it was running but broken [17:42:05] Logged the message, dummy [17:56:09] Amir1: what are you trying to do, and why are you mixing PHP and python? [17:56:58] I did create a system to give users a generator for articles they want to create [17:57:15] *content generator [17:59:04] Amir1: why are you using PHP to call python programs? [17:59:21] Your asking for problems [17:59:43] because python can't be accessed by web [17:59:53] Amir1: wrong [18:00:21] Amir1: I only write python http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag [18:01:16] http://tools.wmflabs.org/betacommand-dev/cgi-bin/ipblock2 [18:01:57] liangent: your instance 'catsort' is hitting virt1000 ldap pretty often. do you have any idea why that is? [18:02:25] Betacommand: ok, let me check and dig deep to see what I can do [18:04:10] Amir1: http://tools.wmflabs.org/betacommand-dev/article_intersection.html is a basic form for a python script [18:08:25] hmm [18:13:04] Coren: do you recognize this request? Can you tell who's making it? (A few labs instances make queries like this now and then, talking to ldap on virt1000 which is deprecated.) https://dpaste.de/DgT0 [18:13:48] Ah, it's autofs, isn't it? [18:14:06] andrewbogott: Yes, that's autofs crap. Whatever instance makes this is certainly already broken (partial, or failed migration from pmtpa) [18:14:28] ok, well, this gives me a good way of detecting which instances still have latent autofs :) [18:14:36] I'll fix 'em up [18:16:07] liangent: nevermind, I think I sorted it out [18:24:17] (03PS1) 10Legoktm: Unbreak redis2stdout [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175472 [18:24:59] (03PS1) 10Legoktm: Fix whitespace in channels.yaml [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175473 [18:25:08] (03CR) 10Legoktm: [C: 032 V: 032] Unbreak redis2stdout [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175472 (owner: 10Legoktm) [18:25:24] (03CR) 10Legoktm: [C: 032 V: 032] Fix whitespace in channels.yaml [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175473 (owner: 10Legoktm) [18:25:54] where can i see the ip range for the tools exec nodes? [18:27:09] (03PS1) 10Legoktm: Send MediaWiki-Core-Team to #mediawiki-core [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175474 [18:27:24] (03CR) 10Legoktm: [C: 032 V: 032] Send MediaWiki-Core-Team to #mediawiki-core [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175474 (owner: 10Legoktm) [18:29:25] !log tools.wikibugs restarting wikibugs for https://gerrit.wikimedia.org/r/175474 [18:29:28] Logged the message, Master [18:33:58] do all tools-exec-* instances have the same public standard ip? [18:34:42] annika_: they shouldnt have a public IP [18:35:14] that's what i meant, but then they still have one [18:35:35] annika_: they do actually [18:35:38] primarily because IRC bots [18:36:02] YuviPanda: are they behind a proxy? [18:36:07] nope [18:36:09] well [18:36:14] firewalled to not accept inbound connections [18:36:30] ah OK [18:36:37] huh, wikitech doesn't show then [18:36:56] some of them don't [18:36:57] https://wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niproject=tools&niregion=eqiad&format=json [18:36:58] has them [18:37:01] annika_: why do you need that info? [18:37:42] i negotiate the blocking of my external links checking bot [18:38:25] annika_: if your getting blocked your doing something wrong [18:39:02] no, i don't, there're site operators that block just everything [18:39:37] annika_: They only block when the traffic from the source is too high [18:39:47] it's 8 links [18:39:54] that's ridiculous [18:40:14] annika_: how often do you check? [18:40:19] every two weeks [18:40:31] are you sending a user agent? [18:40:39] of course [18:40:45] what is it? [18:41:29] and does their robot.txt forbid the links your checking? [18:41:52] German Wikipedia Broken Weblinks Bot; contact: gifti@tools.wmflabs.org [18:42:07] robots.txt doesn't make sense here [18:42:35] annika_: depends on the site owner's preferences [18:42:55] the big databases are excluded anyway [18:42:59] if the site owner doesnt like your robot they may be blocking it [18:43:26] yeah [19:10:42] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Krinkle link https://www.mediawiki.org/w/index.php?diff=1279826 edit summary: [-56] Remove {{Add new Wikimedia engineering project status update}} and fix spacing. [19:44:12] (03PS1) 10Legoktm: Beta-Cluster to #wikimedia-qa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175496 [19:45:07] (03CR) 10Legoktm: [C: 032 V: 032] Beta-Cluster to #wikimedia-qa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175496 (owner: 10Legoktm) [19:50:52] 3Wikimedia-Labs-General: WMFLabs: Delete instance failed to "remove its DNS entry" - https://phabricator.wikimedia.org/T64770#781797 (10Krinkle) [21:28:40] you just hit YCombinator news feed ... https://blog.wikimedia.org/2014/11/24/welcome-to-phabricator-wikimedias-new-collaboration-platform/ [21:29:09] .. in case you were expecting to be lonely on your new service, think again ;-) [21:35:14] andrewbogott: heh, I should setup a 'puppetmaster' and 'ldap' check in shinke too, I think. just got a string of puppet failure notices because you were messing around with ldap [21:35:31] What were the notices? Nothing should've cared [21:35:42] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed when searching for node i-000000cd.eqiad.wmflabs: LDAP Search failed [21:35:49] on tools-master [21:36:03] and a bunch of others [21:36:04] not many tho [21:36:24] Oh, maybe just for a split second while the server restarted? [21:36:29] Everything is happy now, right? [21:36:31] yeah, possibly [21:36:35] yeah [21:36:35] it is [21:36:40] Probably I should've waited a few minutes before the two restarts. [21:36:54] I think that's a valid alert, and my mistake for triggering [21:37:22] well, I think the alert should've been for 'ldap search failed' and not puppet failure [21:37:31] hence need to set up dependencies [21:38:18] so I'll setup puppetmaster on virt1000 and ldap as dependencies for the puppet failures service, and if the puppetmaster fails I'll get one alert about that instead of 1000s about the individual instances [21:39:20] ok, that sounds better [22:08:45] 3Wikimedia-Labs-wikitech-interface: [Regression] WMFLabs: Unable to delete any instance - https://phabricator.wikimedia.org/T72636#782583 (10Krinkle) 5declined>3Resolved a:3Krinkle Yeah, it's worked fine since. It's one of the various side-effects of wikitech seemingly-randomly losing its auth connection w... [22:09:43] 3Wikimedia-Labs-wikitech-interface: [Regression] WMFLabs: Nova project quota broken - https://phabricator.wikimedia.org/T72634#782587 (10Krinkle) 5declined>3Resolved a:3Krinkle It's just another side-effects that re-occurs whenever wikitech user sessions to nova or openstack backend becomes invalid. [22:10:37] (03PS1) 10Greg Grossmeier: Renamed some phab projects, so renaming them here [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175564 [22:14:52] (03PS1) 10Greg Grossmeier: Add #releng to #-qa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175567 [22:16:19] andrewbogott: I'm wondering if I shoult set shinken up to work on debian itself. [22:16:23] andrewbogott: there are 2.0 packages in sid... [22:16:32] so I could perhaps bring those in. [22:16:36] ok, that's crazy. [22:16:43] seems a bit premature :) [22:18:03] yeah [22:18:04] definitely [22:18:08] so ignore that [22:52:31] (03PS1) 10BBlack: dummy test commit [labs/private] - 10https://gerrit.wikimedia.org/r/175581 [22:52:45] (03CR) 10BBlack: [C: 032 V: 032] dummy test commit [labs/private] - 10https://gerrit.wikimedia.org/r/175581 (owner: 10BBlack) [23:45:58] (03PS1) 10Legoktm: Add test_channels_yaml.py to validate the channels.yaml file [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175603 [23:46:42] (03CR) 10Legoktm: [C: 032 V: 032] Add test_channels_yaml.py to validate the channels.yaml file [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/175603 (owner: 10Legoktm)