[04:21:46] 06Labs, 10Labs-Infrastructure, 06Operations, 06Reading-Web-Backlog, and 2 others: https://wikitech.m.wikimedia.org/ serves wikimedia.org portal - https://phabricator.wikimedia.org/T120527#2395633 (10Dzahn) [05:26:53] 06Labs, 10Labs-Infrastructure: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2395655 (10Dzahn) [05:29:39] 06Labs, 10Labs-Infrastructure: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2087511 (10Dzahn) https://wikitech.wikimedia.org/wiki/Add-labs-user#Giving_users_Labs.2Fwikitech.2Fgerrit_access.2C_if_they_already_have_an_SVN_account [05:31:20] 06Labs, 10Labs-Infrastructure: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2395658 (10Dzahn) Hi @BishoyCamel the information requested above is still needed from you for this. Then we can go through the procedure in the link above. [05:34:31] 06Labs, 10Labs-Infrastructure, 06Operations: labs precise and jessie instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#2395660 (10Dzahn) [09:06:17] 06Labs, 10Tool-Labs, 10DBA: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2395904 (10jcrespo) [09:09:31] 06Labs, 10Tool-Labs, 10DBA: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2395919 (10jcrespo) @PeterBowman https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connection_handling_p... [11:40:22] 06Labs, 10Tool-Labs, 10DBA: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2396069 (10PeterBowman) I'm sorry, this shouldn't have happened. I've killed those processes and disabled any scri... [11:49:28] 06Labs, 10Tool-Labs, 10DBA, 06Discovery, 06Maps: Some users use persistent connections that are idle, wasting memory and other resources that could be used for other users - https://phabricator.wikimedia.org/T138283#2396104 (10valhallasw) 10.68.16.70 is maps-wma1.maps.eqiad.wmflabs. [13:22:01] 06Labs, 10Tool-Labs: User tools.admin is in group project-tools - https://phabricator.wikimedia.org/T138182#2396269 (10chasemp) p:05Triage>03Normal [13:22:13] 06Labs, 10Tool-Labs: LDAP entry for group project-tools refers to unknown user account gen-nfs - https://phabricator.wikimedia.org/T138181#2396270 (10chasemp) p:05Triage>03Normal [13:29:41] 06Labs: task not run via crontab - https://phabricator.wikimedia.org/T138178#2396295 (10chasemp) ```tools-bastion-03:/data/project/etwikibots/bots/twitter# python kirjutaks.py Traceback (most recent call last): File "kirjutaks.py", line 5, in from bs4 import BeautifulSoup ImportError: No module na... [13:29:48] 06Labs: task not run via crontab - https://phabricator.wikimedia.org/T138178#2396297 (10chasemp) p:05Triage>03Normal [13:29:56] 06Labs: Document LDAP structure unambiguously - https://phabricator.wikimedia.org/T138151#2396298 (10chasemp) p:05Triage>03Normal [13:30:10] 06Labs: Purge stale data from LDAP - https://phabricator.wikimedia.org/T138150#2396299 (10chasemp) p:05Triage>03Normal [13:31:13] 06Labs, 10Tool-Labs: Install debootstrap, fakechroot and fakeroot on tools - https://phabricator.wikimedia.org/T138138#2396301 (10chasemp) p:05Triage>03Normal can you outline a bit more what you are trying to do exactly? [13:31:23] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries, 10DBA: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2396303 (10chasemp) p:05Triage>03Normal a:05yuvipanda>03None [13:32:33] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool has trouble taking snapshots on OpenStack labs - https://phabricator.wikimedia.org/T138106#2396309 (10chasemp) a:03Andrew @andrew tossing your way as I'm not familiar with the setup here, I will take a look and see if I can ma... [13:32:51] 06Labs: Delete gerrit-dev.wmflabs.org website - https://phabricator.wikimedia.org/T138101#2396311 (10chasemp) p:05Triage>03Normal [13:33:03] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2396312 (10chasemp) p:05Triage>03Normal [13:33:25] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2381540 (10chasemp) seems to be intermittent and I'm not reproducing atm [13:33:28] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries, 10DBA: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2396315 (10jcrespo) I was wrong, you are executing 1000 times a 32-million row query (but the conclusions still stand). [13:33:36] 06Labs, 10Labs-Infrastructure: Delete staging labs project - https://phabricator.wikimedia.org/T137726#2396316 (10chasemp) p:05Triage>03Normal [13:34:49] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2396329 (10chasemp) p:05Triage>03Normal [13:35:10] 06Labs, 10Labs-Infrastructure: Create a service hostname for labs statsd - https://phabricator.wikimedia.org/T136968#2396331 (10chasemp) p:05Triage>03Normal [13:35:16] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Special page reports not updating on Wikitech - https://phabricator.wikimedia.org/T136926#2396332 (10chasemp) p:05Triage>03Normal [13:36:29] 06Labs, 10Labs-Infrastructure, 10Horizon: Disassociating floating IP does not show it as available in the horizon project quota summary - https://phabricator.wikimedia.org/T136872#2396361 (10chasemp) p:05Triage>03High [13:37:29] 06Labs, 10Labs-Infrastructure: Empty default security group for newly created project - https://phabricator.wikimedia.org/T136871#2396363 (10chasemp) p:05Triage>03High @bd808 is this still happening? [13:39:10] 06Labs, 10DBA: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2396380 (10chasemp) p:05Triage>03High [13:42:34] 06Labs, 10Tool-Labs: "webservice" command (with jsub) doesn't work in crontab - https://phabricator.wikimedia.org/T135348#2396416 (10chasemp) p:05Triage>03Low [13:43:00] 06Labs, 10DBA: Missing data on labs replica database - https://phabricator.wikimedia.org/T133715#2396434 (10chasemp) p:05Triage>03Normal [13:47:14] 06Labs, 10DBA: Discrepancy between labsdb replicas of arwiki_p.user_groups - https://phabricator.wikimedia.org/T133469#2396475 (10chasemp) p:05Triage>03Normal [13:47:43] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2396476 (10chasemp) >>! In T132828#2343303, @Krenair wrote: >>>! In T132828#2341950, @chasemp wrote: >> @krenair can you elaborate on who this indicates: >> >>> then the task is... [13:47:49] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2396477 (10chasemp) p:05Triage>03Normal [13:49:05] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2396500 (10chasemp) p:05Triage>03Normal one thought is we have an influx of new labsdb things coming I believe. This way sort itself out w/o a lot of in-place shuffling. [14:11:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Timmoody was created, changed by Timmoody link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Timmoody edit summary: Created page with "{{Tools Access Request |Justification=Test algorithms for creating slices of wikipedia for use as zim file. |Completed=false |User Name=Timmoody }}" [15:01:13] 06Labs, 10Tool-Labs: LDAP entry for group project-tools refers to unknown user account gen-nfs - https://phabricator.wikimedia.org/T138181#2396704 (10zhuyifei1999) https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Titleblacklist&diff=next&oldid=173726 [15:28:23] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2396760 (10Krenair) Can we alter the source code and configuration files of the bot? [15:35:59] 06Labs: task not run via crontab - https://phabricator.wikimedia.org/T138178#2396768 (10bd808) >>! In T138178#2396295, @chasemp wrote: > ```tools-bastion-03:/data/project/etwikibots/bots/twitter# python kirjutaks.py > Traceback (most recent call last): > File "kirjutaks.py", line 5, in > from bs4... [15:46:04] 06Labs: task not run via crontab - https://phabricator.wikimedia.org/T138178#2392407 (10valhallasw) Last write to the .err file was may 9th ``` tools.etwikibots@tools-bastion-03:~$ ls -l /data/project/etwikibots/log/kirjutaks.err -rw-rw---- 1 tools.etwikibots tools.etwikibots 317340 May 9 10:00 /data/project/e... [16:02:02] 06Labs: task not run via crontab - https://phabricator.wikimedia.org/T138178#2396858 (10bd808) This is caused by the jsub rewrite. ``` $ JOBUTILS_QSUB=echo jsub /data/project/etwikibots/bin/kirjutaks.sh [Tue Jun 21 15:56:37 2016] Error options embedded in /data/project/etwikibots/bin/kirjutaks.sh: argument -l:... [16:18:30] (03PS1) 10BryanDavis: jsub: Strip trailing comments from jsubrc and embedded options [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295361 (https://phabricator.wikimedia.org/T138178) [16:24:00] (03PS2) 10BryanDavis: jsub: Update jsubrc and embedded options parsing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295361 (https://phabricator.wikimedia.org/T138178) [16:24:49] (03CR) 10BryanDavis: jsub: Update jsubrc and embedded options parsing (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295361 (https://phabricator.wikimedia.org/T138178) (owner: 10BryanDavis) [16:25:35] valhallasw`cloud, chasemp: I think that ^ will fix T138178 [16:25:35] T138178: task not run via crontab - https://phabricator.wikimedia.org/T138178 [16:28:18] bd808: ahh nice, I thought maybe virtualenv but was too scattered brained [16:28:48] bd808: alternatively we can change the .sh files [16:29:18] meh. let's just put back the small bit of crappy behavior [16:29:56] yeah this is possibly not the only affected just the only noticed so far etc [16:30:01] the killing off of script comments and jsubrc is a bigger task that we haven't startedt to work on yet [16:30:13] I'd prefer line = line.split('#')[0], but this also works [16:30:34] (I *think* the regex stops at the first #?) [16:30:40] (03CR) 10Merlijn van Deen: [C: 031] jsub: Update jsubrc and embedded options parsing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295361 (https://phabricator.wikimedia.org/T138178) (owner: 10BryanDavis) [16:31:07] wait, I can just +2 :-p [16:31:08] yeah, it stops at the first # or whitespace in the argument [16:31:12] (03CR) 10Merlijn van Deen: [C: 032] jsub: Update jsubrc and embedded options parsing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295361 (https://phabricator.wikimedia.org/T138178) (owner: 10BryanDavis) [16:31:29] which is nearly what the old one did [16:31:46] (03Merged) 10jenkins-bot: jsub: Update jsubrc and embedded options parsing [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295361 (https://phabricator.wikimedia.org/T138178) (owner: 10BryanDavis) [16:31:59] the perl regex only cared about #, space, newline and tab [16:32:29] ah [16:33:43] For indescribable reasons I hate seeing regexs that treat [ \n\t] differently than \s [16:34:33] probably mostly because of unicode and cut-n-paste word processors causing havoc in past projects [16:37:14] I'll build a new jobutils deb [16:39:47] (03PS1) 10BryanDavis: New jobutils release [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295363 [16:40:15] (03CR) 10BryanDavis: [C: 032] New jobutils release [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295363 (owner: 10BryanDavis) [16:40:46] (03Merged) 10jenkins-bot: New jobutils release [labs/toollabs] - 10https://gerrit.wikimedia.org/r/295363 (owner: 10BryanDavis) [16:49:34] !log tools Updated jobutils to v1.14 for T138178 [16:49:35] T138178: task not run via crontab - https://phabricator.wikimedia.org/T138178 [16:49:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:52:11] 06Labs, 10Tool-Labs, 15User-bd808: task not run via crontab - https://phabricator.wikimedia.org/T138178#2397021 (10bd808) a:03bd808 @WikedKentaur I'd appreciate it if you could verify that the update to `jsub` fixes your job execution problem. [16:56:03] 06Labs, 10Labs-Infrastructure: Empty default security group for newly created project - https://phabricator.wikimedia.org/T136871#2397052 (10bd808) Yes. I just created a new project named [[https://wikitech.wikimedia.org/wiki/Nova_Resource:T136871|t136871]] and it has the same empty default security group. [17:28:48] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool has trouble taking snapshots on OpenStack labs - https://phabricator.wikimedia.org/T138106#2397243 (10Andrew) Most likely this is failing due to lack of resources on the virt nodes (that's my go-to answer for everything this mon... [17:30:33] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2397254 (10Andrew) [18:35:29] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: adywiki is missing the associated adywiki_p database with appropriate views - https://phabricator.wikimedia.org/T135029#2286195 (10Gehel) [18:56:13] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2397393 (10MaxSem) [19:33:26] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Timmoody was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=670950 edit summary: [19:45:29] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2397500 (10tom29739) That project is no longer needed. [19:46:09] 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2397504 (10tom29739) [19:46:12] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2397505 (10tom29739) [19:46:14] 06Labs: Create new labs project tools-xdebug-testing - https://phabricator.wikimedia.org/T138097#2397501 (10tom29739) 05Open>03Resolved a:03tom29739 No longer needed. [20:15:02] 06Labs, 10Tool-Labs: Create new labs project labs-captcha - https://phabricator.wikimedia.org/T138348#2397573 (10tom29739) [20:38:40] bd808, can you create this labs project for me? https://phabricator.wikimedia.org/T138348 Thanks. [20:41:29] tom29739: is "labs-captcha" a good name for the project? I assume you are intending to make some privacy conscious captcha service. [20:42:46] which is neat, but having "labs" in the project name seems possibly confusing for folks [20:43:04] Maybe just 'captcha' then? [20:43:43] (not to barge in or anything) you're talking about creating a project named 'captcha'? [20:43:48] bd808: This project is a privpol compliant captcha service. [20:43:55] Yeah. [20:43:57] enterprisey: Yeah. [20:44:11] sounds like a tag to me, not a project working on one specific thing [20:44:31] (unless the project is meant to cover everything involving captchas) [20:44:39] its not a phab thing, it's a Labs project [20:44:46] ^ [20:47:13] It feels easily confusable though [20:47:31] tom-captcha? [20:47:53] not-recaptcha? [20:47:56] Opencaptcha? [20:48:12] open-captcha seems good. [20:49:12] That's not as confusable as just captcha. [20:49:21] bd808: There's a GeSHi joke to be made somewhere in here [20:49:31] http://www.opencaptcha.com/ [20:49:46] oh geshi [20:49:54] I was more meaning our extension name [20:49:57] privpol-captcha [20:50:32] I can't see anything that conflicts with that. [20:51:27] works for me. [20:52:06] tom29739: you should think about using Puppet to setup all the bits you need too. It will make maintaining things much easier for you [20:52:33] and if you screw everything over royally, potentially minimal work to get it back up again [20:52:45] s/if/when/ [20:53:04] It's going to happen sometime, it always does. [20:53:59] I was trying to be nice [20:54:18] * bd808 eyes Reedy suspiciously [20:54:28] I always manage to screw things up. [20:55:03] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2397705 (10bd808) [20:55:05] 06Labs, 10Tool-Labs, 15User-bd808: Create new labs project labs-captcha - https://phabricator.wikimedia.org/T138348#2397702 (10bd808) 05Open>03Resolved a:03bd808 https://wikitech.wikimedia.org/wiki/Nova_Resource:Privpol-captcha [20:55:08] {{done}} [20:56:49] legoktm: around? [20:56:50] 06Labs, 10Tool-Labs, 15User-bd808: Create new labs project privpol-captcha - https://phabricator.wikimedia.org/T138348#2397721 (10Reedy) [21:00:26] 06Labs, 10Labs-Infrastructure: Empty default security group for newly created project - https://phabricator.wikimedia.org/T136871#2397727 (10bd808) Now I have created the privpol-captcha project and its default security group is fine. A noticable difference from when I created the t136871 project is that my N... [21:05:29] (03CR) 10Lokal Profil: [C: 032] "First merge of the Hackathon!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 (owner: 10Jean-Frédéric) [21:06:26] (03Merged) 10jenkins-bot: Add CSS linting via stylelint [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/293635 (owner: 10Jean-Frédéric) [21:15:06] 06Labs, 10Tool-Labs, 13Patch-For-Review: Figure out a way to keep MerlBot running when the HTTP POST loophole is closed - https://phabricator.wikimedia.org/T121279#2397756 (10bd808) >>! In T121279#2392449, @Merl wrote: > Hi, i have tried to read a lot of discussions to get back to the current state of discus... [21:58:48] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2397893 (10Gehel) Summary of a discussion with @ori: The maintain-replicas script creates a new sch... [22:37:11] I'm following these docs: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster it says 'Now, go back to the instance configuration page and enable the role::puppet::self class.' How do I do that, I can't see any options in Horizon or Wikitech. [22:37:56] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 07Blocked-on-Operations: adywiki and jamwiki are missing the associated *_p databases with appropriate views - https://phabricator.wikimedia.org/T135029#2397950 (10Krenair) I agree, although it needs to be a separate ticket and I don't think we can just... [22:40:26] hello, is graphite completely kaput? [22:44:27] ahh I see the relevant issue on phabricator :) [22:46:29] tom29739: you can't see it on this page? -- https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&instanceid=0a1d407b-2eb0-47c9-b1ac-c3a5da8afaa3&project=privpol-captcha®ion=eqiad [22:46:57] 'The specified resource does not exist.' [22:47:23] hmmm... maybe your Nova session has timedout/broken [22:47:27] I get nothing in Special:NovaInstance too. [22:47:55] yeah, log out of wikitech and log back in. This is a lame bug that has been open and mysterious for years [22:48:02] I've deleted Nova_Resource:Puppet.privpol-captcha.eqiad.wmflabs but it still shows up. [22:48:43] I see 5 instances for the privpol-captcha project at https://wikitech.wikimedia.org/wiki/Special:NovaInstance [22:48:51] I see them now. [22:49:34] On here, it shows 6, no matter how much I purge it: https://wikitech.wikimedia.org/wiki/Nova_Resource:Privpol-captcha [22:51:52] tom29739: that's weird. The instance page still exists but it's not showing in the wikitech or horizon dashboards for me [22:52:11] it's like it got half cleaned up [22:52:24] maybe just the SMW data is left laying around? [22:52:43] I terminated the instance in horizon. [22:53:19] I terminated a couple other instances too, and they went away properly. [22:53:41] Id' say open a bug about it and then just get on with your life. :) [22:54:34] I'm sort of amazed you were allowed to make a host with the name "puppet" [22:56:10] I made it by mistake. [22:56:32] I thought it would append the project name on the end, but it didn't :D [22:56:58] yeah. the dns setup is a bit goofy [22:57:26] the really confusing thing is that the project name part of the dns name is optional. [22:58:37] captcha-web-2.eqiad.wmflabs. == captcha-web-2.privpol-captcha.eqiad.wmflabs. [23:30:47] bd808: mind if I use this IRC conversation in this bug report? [23:46:19] !log install elasticsearch to deployment-logstash3 and point logstash on deployment-logstash2 to duplicate inserts to 03 as well [23:46:19] install is not a valid project. [23:46:27] !log logstash install elasticsearch to deployment-logstash3 and point logstash on deployment-logstash2 to duplicate inserts to 03 as well [23:46:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Logstash/SAL, Master