[00:34:26] 06Labs, 10Labs-Infrastructure, 06Operations: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2435798 (10AlexMonk-WMF) ``` 121.16.68.10.in-addr.arpa domain name pointer ci-jessie-wikimedia-47938.contintcloud.eqiad.wmflabs. 121.16.68.10.in... [01:14:09] 06Labs: Access needed to mwui.wmflabs.org - https://phabricator.wikimedia.org/T123316#1926132 (10AlexMonk-WMF) @Volker_E was added back in January: https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Editor-engagement&diff=prev&oldid=254862 Is it time to close this? [01:23:17] 06Labs, 10Labs-Infrastructure: Automatically updated list of all configured domains - https://phabricator.wikimedia.org/T45580#486909 (10AlexMonk-WMF) So it'd essentially just be a dump of Designate's data for the wmflabs.org DNS zone? [07:11:09] !log bots Restarted wm-bot, RSS feed error again [07:11:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/SAL, Master [09:43:23] 06Labs, 10Labs-Infrastructure, 06Operations: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2436546 (10hashar) Until the DNS leak is identified entries will keep leaking. It is quite easy to retrieve all of them from the Designate database, so there is no... [09:44:38] (03PS4) 10Lokal Profil: Fix ID dump process and tools [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [09:45:24] (03CR) 10Lokal Profil: Fix ID dump process and tools (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [09:45:44] (03CR) 10jenkins-bot: [V: 04-1] Fix ID dump process and tools [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [09:47:20] (03CR) 10Lokal Profil: "Live changes. how can something so wrong feel so right ;)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [09:56:52] !log zuul, gerrit and jenkins are all setup now (Thanks hashar for helping me) [09:56:53] zuul, is not a valid project. [09:57:02] !log phabricator zuul, gerrit and jenkins are all setup now (Thanks hashar for helping me) [09:57:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, Master [09:57:10] hashar ^^ [09:57:11] :) [10:05:15] (03CR) 10Lokal Profil: "Since the 18n issue looks similar to that on the SearchPage I added some debugging to the end of the output on the live page." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [10:21:36] (03CR) 10Lokal Profil: "Lets ignore the i18n issue for now. That needs to be dealt with (via T139267) for a much larger set of files." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [10:27:20] (03CR) 10Lokal Profil: "The failing test is unrelated and due to T139580" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [10:59:49] (03PS1) 10Lokal Profil: Add more data to primkey warning [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 [11:00:22] (03PS2) 10Lokal Profil: Add more data to primkey warning [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 (https://phabricator.wikimedia.org/T138633) [11:01:20] (03CR) 10jenkins-bot: [V: 04-1] Add more data to primkey warning [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:02:46] (03CR) 10Lokal Profil: "The failing test is unrelated and due to T139580" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:08:20] (03PS1) 10Lokal Profil: Revert accidentally overwritten updates to i18n/qqq.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297768 (https://phabricator.wikimedia.org/T139580) [11:27:17] (03CR) 10Nikerabbit: [C: 031] Revert accidentally overwritten updates to i18n/qqq.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297768 (https://phabricator.wikimedia.org/T139580) (owner: 10Lokal Profil) [11:30:23] (03CR) 10Jean-Frédéric: [C: 032] Revert accidentally overwritten updates to i18n/qqq.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297768 (https://phabricator.wikimedia.org/T139580) (owner: 10Lokal Profil) [11:31:25] (03Merged) 10jenkins-bot: Revert accidentally overwritten updates to i18n/qqq.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297768 (https://phabricator.wikimedia.org/T139580) (owner: 10Lokal Profil) [11:38:45] (03PS3) 10Jean-Frédéric: Add more data to primkey warning [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:38:53] mutante I see jessie-bastion-01 and -02 in the project 'ganglia' that you created. do you still need those? [11:39:33] (03CR) 10Jean-Frédéric: "I rebased the change after the merge of Ib708bc19a1e0d82c59035332f021f28787ac2d75" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:41:39] (03CR) 10Jean-Frédéric: [C: 032] Add more data to primkey warning [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:42:42] (03Merged) 10jenkins-bot: Add more data to primkey warning [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297766 (https://phabricator.wikimedia.org/T138633) (owner: 10Lokal Profil) [11:42:54] (03PS5) 10Lokal Profil: Fix ID dump process and tools [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [11:44:03] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Setup monitoring for kubernetes core components. - https://phabricator.wikimedia.org/T131929#2436953 (10yuvipanda) This will check for the webservice to start and stop, which is exercising the following things; 1. Master is reachable and responsi... [11:44:56] 06Labs, 06Operations: labvirt1011 periodically unavailable - https://phabricator.wikimedia.org/T139555#2436959 (10Andrew) [11:46:09] (03CR) 10Lokal Profil: "I rebased the change after the merge of Ib708bc19a1e0d82c59035332f021f28787ac2d75 to make CI happy" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [11:46:11] andrewbogott btw, none of the tools instances on labvirt1011 are critical - there's a couple of exec nodes and a spare static node. [11:46:13] just fyi [11:46:55] yuvipanda: good to know! If you have the time to depool/drain/delete everything that you can from there that would help prepare us for the inevitable. [11:47:26] (but let me know if/when you're doing that so we can keep track of what's happening during troubleshooting) [11:48:13] andrewbogott ok! We *can* just delete the xlarge tools-web-static-02, but that just means no redundancy for that service (tools-static) for a bit. But it's super easy to recreate... [11:48:45] yuvipanda: if it's inactive then it doesn't matter [11:48:55] yup [11:48:57] ok [11:49:11] What I mean about depool/drain is that we should minimize the impact on tools from future reboots/shutdowns of the system [11:51:31] ok [12:29:42] 06Labs, 06Operations: labvirt1011 periodically unavailable - https://phabricator.wikimedia.org/T139555#2437159 (10Andrew) more background: I did a dist-upgrade on that system right before putting it into service. That was on 2016-06-26. The system behaved well until 2016-06-05 when alarms started firing all... [12:45:19] !log tools start deployment of k8s 1.3.0wmf4 for T139259 [12:45:20] T139259: Upgrade to Kubernetes 1.3 - https://phabricator.wikimedia.org/T139259 [12:45:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [12:51:41] 06Labs, 06Operations: labvirt1011 periodically unavailable - https://phabricator.wikimedia.org/T139555#2437231 (10Andrew) This is almost certainly fixed by https://gerrit.wikimedia.org/r/#/c/297783/ we'll know soon enough. [13:07:33] 06Labs, 06Operations: labvirt1011 periodically unavailable - https://phabricator.wikimedia.org/T139555#2437284 (10Andrew) 05Open>03Resolved a:03Andrew So here's the story: - A typo in dhcpd cofig which resulted in 1012 1013 and 1014 wanting the same IP as 1011 - This shouldn't have mattered since those... [13:09:50] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Drop some Trusty permanent slaves from integration labs project - https://phabricator.wikimedia.org/T139535#2437291 (10hashar) a:03hashar The Jenkins graph above is average so it does not accomodate for spikes :( I created some more g... [13:37:10] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Drop some Trusty permanent slaves from integration labs project - https://phabricator.wikimedia.org/T139535#2437394 (10hashar) They were both on labvirt1010 which recovered 16GBytes of memory :-] {F4249228 size=full} [15:00:48] (03CR) 10Jean-Frédéric: [C: 032] Fix ID dump process and tools [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [15:02:06] (03Merged) 10jenkins-bot: Fix ID dump process and tools [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/297526 (owner: 10Jean-Frédéric) [15:22:25] 06Labs, 10Tool-Labs: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437677 (10jcrespo) [15:23:06] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437691 (10jcrespo) [15:23:14] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437677 (10yuvipanda) @Magnus since I think he wrote petscan? [15:34:21] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Upgrade to Kubernetes 1.3 - https://phabricator.wikimedia.org/T139259#2437717 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Upgrade complete! [16:00:47] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437810 (10Magnus) The five threads are for bot accounts only. Normal user accounts get single thread with delay. I have used my own bot account //a lot// over the years, with previo... [16:07:24] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437819 (10jcrespo) @Magnus, as you can see on the discussion I agreed with you initially, and in no way I am giving you any responsibility for this particular incident. However, the... [16:33:57] yuvipanda: no, i don't need jessie-bastion-01 and -02 anymore. i dont recall that, technically it could have been another project admin making them [16:34:17] mutante nova show showed me your name on them [16:34:24] ok, delete them :) [16:34:35] it was about testing if ganglia aggregators work on jessie [16:34:42] which we have in prod now [17:18:37] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438159 (10Magnus) Just run it as a normal user and not a bot user! [17:21:11] Hi, can anyone help me? [17:21:47] I'm trying to do OAuth Consumer registration for my bot, but it keeps saying email has not been confirmed, even if it has. [17:25:10] Raystorm: where are you doing this? [17:25:19] meta [17:25:54] and what's the exact error message you get? [17:26:17] "Your account email address has not yet been confirmed" [17:29:38] Raystorm, what does it say in https://meta.wikimedia.org/wiki/Special:Preferences at 'Email confirmation:'? [17:29:52] Email was confirmed a month ago [17:31:41] Raystorm: try changing the email address to another one and back? [17:32:12] Did. No luck :( I have been trying for a week. I am out of options [17:32:53] Are you using the same email on the form that's been confirmed on your account? [17:33:02] yes [17:33:23] Weird. [17:33:30] Right? [17:33:44] Hola Raystorm :) [17:33:51] yeah, that check comes after the 'is user email confirmed' checked [17:33:54] CristianCantoro :D [17:35:11] Raystorm: and meta happily lets me send you an email. What the heck? [17:35:45] Yeah, got it. And yet can't do OAuth, argh! [17:36:14] Raystorm: this is above my pay grade ;-) could you file a bug in phabricator? [17:37:06] I don't even know what to say. Help, it says email is not confirmed but it is? [17:37:46] Raystorm: and yet, that's basically the bug [17:37:52] 'I'm trying to register an oauth consumer but it says my email is not confirmed. It is confirmed; I tried switching email addresses, and other people can send me email via Special:SendEmail' [17:39:10] Ah but wait [17:39:18] Send me the email to the RaystormBot account [17:39:42] You sent it to Raystorm. Let's see if I can get the email with the bot account [17:41:06] Raystorm, it says This user has not specified a valid email address. [17:41:18] But I have :( [17:41:31] CristianCantoro saw the screencap [17:41:39] Raystorm: are you sure you haven't switched around the email addresses for User:Raystorm instead? [17:41:52] I'm sure :( [17:42:09] Raystorm: what? [17:42:33] CristianCantoro I'm saying that you saw the screencap of the "Email has been confirmed" [17:42:57] Raystorm: yup, I am not denying the bug :) [17:43:05] You are my witness xD [17:45:22] I'm going to report this on phabricator [17:48:29] Raystorm: :) [17:52:45] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438321 (10jcrespo) I will see what the user responds, and act depending on it. [17:55:27] Done: https://phabricator.wikimedia.org/T139633 [17:58:11] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438354 (10Bugreporter) Running it as a normal user will flood recent change, See https://www.wikidata.org/wiki/Wikidata:Administrators%27_noticeboard/Archive/2014/05#Flooding_of_Spe... [18:03:37] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438377 (10Bugreporter) I try to limit the negative effect of running the tools. At the beginning at most 6-7 tabs are runs. Then I keep only one tab after warning. Now I use two ta... [18:04:42] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437677 (10Bugreporter) oops this user is not active at phabricator. [18:08:36] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438413 (10Magnus) " 6-7 tabs are run" I believe we found the root problem :-) [18:10:24] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438422 (10Bugreporter) This is previously, as I didn't know how many tabs should be run at most, and what problem would occur when too many tabs are running. [18:10:31] well, I am off. Hopefully someone will see that bug report soonish. Thanks for the help valhallasw` tom29739 CristianCantoro :) [18:10:37] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438423 (10jcrespo) I think we agreed to use only one "tab" at a time to follow API:Etiquette. I will block all your queries if they continue producing errors in the next 10 minutes,... [18:13:01] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2438430 (10Bugreporter) Currently (and since 20+ minute ago) only one is running. [18:15:20] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is running too fast for Wikidata - https://phabricator.wikimedia.org/T139618#2437677 (10jcrespo) 05Open>03Resolved a:03jcrespo Thank you. I see lower amount of errors in the last 20 minutes. I will be monitoring the logs in case the errors return. [18:22:06] 06Labs, 10Tool-Labs, 10DBA, 10Wikidata: Petscan is being used with excesive parallelism by a user on Wikidata - https://phabricator.wikimedia.org/T139618#2438527 (10jcrespo) [18:40:17] 06Labs, 10Labs-Infrastructure: Automatically updated list of all configured domains - https://phabricator.wikimedia.org/T45580#2438558 (10AlexMonk-WMF) We have an effectively manually-updated, maybe-one-off page showing these domains here: https://wikitech.wikimedia.org/wiki/Purge_2016 It was generated using h... [19:57:17] 06Labs, 10Tool-Labs, 10puppet-compiler: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#2438903 (10valhallasw) Failing hiera was because I forgot to set `RUBYLIB=/mnt/jenkins-workspace/puppet-compiler/1467917450/production/src/modules/wmflib/lib/` , which meant... [20:24:57] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2439038 (10mmodell) a:05mmodell>03demon Since you're working on the phab puppet stuff [20:35:12] 06Labs, 10Labs-Sprint-102, 10Labs-Sprint-103, 10Labs-Sprint-104, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1360119 (10AlexMonk-WMF) https://wikitech.wikimedia.org/wiki/Recover_instance_from_NFS may be helpful to some, although it... [20:40:23] 06Labs, 10Labs-Infrastructure, 06Operations: Depleted connection tracking table on labvirt1010 - https://phabricator.wikimedia.org/T139598#2439126 (10Andrew) Related: https://openstack-in-production.blogspot.com/2015/01/exceeding-tracked-connections.html [20:45:44] 06Labs, 10Labs-Infrastructure, 06Operations, 13Patch-For-Review: Depleted connection tracking table on labvirt1010 - https://phabricator.wikimedia.org/T139598#2439142 (10chasemp) p:05Triage>03High [22:22:57] !log tools.heritage Deployed latest from Git: 6e6cc59 [22:23:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [22:26:31] !log tools.heritage Deployed latest from Git: 1022e80, ad29828, 7ab4bcf, 48c96b4 (T139580), f29995d (T138633), 92f9234 [22:26:33] T138633: improve logging - https://phabricator.wikimedia.org/T138633 [22:26:33] T139580: translatewiki not pulling i18n updates from heritage - https://phabricator.wikimedia.org/T139580 [22:26:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [22:40:12] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2439950 (10greg) [23:04:28] 06Labs, 07Tracking: Create labtest cluster (tracking) - https://phabricator.wikimedia.org/T120293#2440061 (10Andrew) [23:04:30] 06Labs: Install and configure labtestservices2001 - https://phabricator.wikimedia.org/T120300#2440059 (10Andrew) 05Open>03Resolved a:03Andrew [23:04:35] 06Labs, 07Tracking: Create labtest cluster (tracking) - https://phabricator.wikimedia.org/T120293#1850359 (10Andrew) [23:06:18] 06Labs: Install and configure labtestvirt2001 as a nova-compute host - https://phabricator.wikimedia.org/T120296#2440068 (10Andrew) [23:06:25] 06Labs, 07Tracking: Create labtest cluster (tracking) - https://phabricator.wikimedia.org/T120293#1850359 (10Andrew) [23:06:27] 06Labs: Install and configure labtestvirt2001 as a nova-compute host - https://phabricator.wikimedia.org/T120296#1850409 (10Andrew) 05Open>03Resolved a:03Andrew [23:57:56] 10Tool-Labs-tools-Global-user-contributions: Limit list to contributions made within a stated date range - https://phabricator.wikimedia.org/T139702#2440229 (10Whatamidoing-WMF)