[00:01:28] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2340013 (10yuvipanda) [00:01:30] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Allow accessing kubernetes services / apiserver from bastions - https://phabricator.wikimedia.org/T136413#2340011 (10yuvipanda) 05Open>03Resolved Done now, kubectl works from both of the bastions. [00:01:52] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2101957 (10yuvipanda) [00:01:55] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Allow accessing kubernetes services / apiserver from bastions - https://phabricator.wikimedia.org/T136413#2340014 (10yuvipanda) 05Resolved>03Open Re-opening since I didn't document this yet. [00:44:50] !log openstack reboot labs-dynamicproxy-test to get it out of an NFS handle induced funk [00:44:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL, Master [00:56:41] Hi, could someone restart the "phabricator-bug-status" please? Instructions (copied from talkpage) are: [00:56:43] ssh tools-login.eqiad.wmflabs [00:56:43] become phabricator-bug-status [00:56:43] webservice2 uwsgi-python restart [00:56:57] 10Labs-project-extdist: 404 from /dist/extensions/VisualEditor-REL1_23-9883566.tar.gz - https://phabricator.wikimedia.org/T110031#2340053 (10Pokefan95) p:05Triage>03Unbreak! Matching priority with T136564 [00:57:23] 10Labs-project-extdist: extdist tarball generator is erroring on VisualEditor REL1_23 - https://phabricator.wikimedia.org/T121748#2340059 (10Pokefan95) p:05Triage>03Unbreak! Matching priority with T136564 [00:58:33] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2339090 (10Pokefan95) Downloading a screenshot of the MobileFrontend extension works for MediaWiki versions 1.25 and below, but not 1.26 and master. [01:02:43] quiddity: done, can you link me to to the page with those instructions? [01:03:15] YuviPanda, <3 - and I just happened to remember that they were in https://en.wikipedia.org/wiki/MediaWiki_talk:Gadget-BugStatusUpdate.js#Not_working [01:04:26] hmmm, still not working. Must be something more complicated than the usual. :< [01:04:33] YuviPanda, ^ [01:04:41] I'll ask Matt, tomorrow. [01:05:04] quiddity: ok! [03:02:54] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Goal: Allow using k8s instead of GridEngine as a backend for webservices (tracking) - https://phabricator.wikimedia.org/T129309#2340099 (10yuvipanda) [03:02:56] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Allow accessing kubernetes services / apiserver from bastions - https://phabricator.wikimedia.org/T136413#2340097 (10yuvipanda) 05Open>03Resolved And documented https://wikitech.wikimedia.org/wiki/Tools_Kubernetes#Bastion_nodes [03:57:12] 06Labs, 10Wikimedia-Bugzilla: Add a link to the Phabricator task for bugs on bugs.wmflabs.org - https://phabricator.wikimedia.org/T109840#2340137 (10Peachey88) [04:02:52] musikanimal: Yo. [04:03:12] Hey [04:04:02] You want to join -xtools? [04:10:14] Sure [04:31:46] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/PetrohsW was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=594685 edit summary: [04:57:38] Krinkle: no clue never involved with either [05:55:01] 10Labs-project-extdist: extdist tarball generator is erroring on VisualEditor REL1_23 - https://phabricator.wikimedia.org/T121748#2340225 (10Legoktm) 05duplicate>03Open p:05Unbreak!>03Triage Please don't close bugs as duplicates unless you know they're actually duplicates. [06:12:32] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [06:14:30] PROBLEM - Puppet run on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:14:41] hmm [06:19:42] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:21:01] _joe_: ^ a bunch of machines are reporting failures tho [06:21:05] and these are lagged usually [06:23:15] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:24:17] <_joe_> YuviPanda: I think the problem is you lack /etc/default/grub [06:24:37] <_joe_> which, on a trusty machine, is bananas [06:24:40] hmm, tools-redis is actually jessie [06:24:48] * YuviPanda checks [06:24:57] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:25:23] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:25:28] _joe_: interesting, the jessie redis one had a totally different 'Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Augeas[grub2] is already declared in file /etc/puppet/modules/base/manifests/grub.pp:22; cannot redeclare at /etc/puppet/modules/base/manifests/labs.pp:34 on node tools-redis-1001.tools.eqiad.wmflabs [06:25:29] <_joe_> YuviPanda: it seems you don't install grub there [06:25:30] ' [06:25:44] <_joe_> YuviPanda: argh [06:25:55] <_joe_> let's fix it [06:26:01] * YuviPanda checks other jessies [06:28:15] <_joe_> YuviPanda: this is yet another case of a bad series of puppet changes [06:28:29] <_joe_> YuviPanda: it will take me some time to fix this properly [06:28:36] yup this is actually failing on all jessies as well [06:28:38] <_joe_> are you willing to wait ~ 1 hour? [06:28:47] _joe_: yup [06:28:50] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:28:59] <_joe_> YuviPanda: so AFAICT the trusty image you pointed me to has no grub installed [06:29:09] yeah, I'm now going to check if that's all the same images though. [06:29:15] I do know they come back up after reboots [06:29:41] actually, this might be affecting *all* trusty machines too, I'll know in a minute [06:29:51] (the tools puppetmaster hadn't fully updated when I ran it on a few machines) [06:31:29] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:33:19] PROBLEM - Puppet run on temp-test-trusty-package is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:34:07] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:34:29] _joe_: from what I can see, this is breaking puppet on *all* trusty instances, rather than just a specific image. [06:35:16] I just checked 3 different instances created in totally different times with totally different base images and all fail with same thing [06:35:29] <_joe_> YuviPanda: yes, I checked it, and I am fixing it now [06:35:36] <_joe_> btw it's not really breaking puppet [06:35:42] <_joe_> just the augeas call fails [06:35:50] <_joe_> it's honestly a pity [06:36:00] <_joe_> I'll have to dig deeper later today [06:36:03] _joe_: right, fair, but it's going to flood my inbox :D broken windows etc. [06:36:07] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:36:09] (with puppet fail alerts) [06:36:23] I'll stop pinging you and let you fix it now :) [06:36:35] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:37:48] <_joe_> YuviPanda: https://gerrit.wikimedia.org/r/#/c/291772/ [06:38:33] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:38:49] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [06:39:17] _joe_: do you want me to help test it? [06:39:47] <_joe_> YuviPanda: if you can cherry-pick to tools puppetmaster, that's great [06:40:30] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:41:56] _joe_: yeah doing now [06:42:16] <_joe_> ack me when you cherry-picked [06:42:36] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:42:48] PROBLEM - Puppet run on tools-prometheus-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:42:53] <_joe_> btw, again, adding things that duplicate functionality to labs only is a bit sad [06:42:58] <_joe_> and we should not do it [06:44:01] <_joe_> uhm almost there [06:44:04] <_joe_> un-cherry-pick [06:44:06] <_joe_> :) [06:44:26] <_joe_> no sorry, keep it [06:45:12] _joe_: that does make the fails go away [06:45:14] <_joe_> uhm which version did you cherry-pick? [06:45:27] PS [06:45:29] 3 [06:45:41] <_joe_> nah, ps 4 [06:45:42] <_joe_> :P [06:45:56] <_joe_> and I need to add a specific transitional thing to the labs manifest [06:46:06] <_joe_> so give me 1 min and pick ps 5 [06:47:18] sure [06:48:14] <_joe_> pick it :) [06:50:09] <_joe_> did you un-cherrypicked it? [06:52:09] _joe_: I cherry picked 5 [06:52:33] <_joe_> ok on jessies it does the right thing [06:52:39] <_joe_> not sure about trustys [06:53:45] <_joe_> are you running puppet on tools-exec-1409? [06:53:53] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [06:53:53] PROBLEM - Puppet run on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:55:19] <_joe_> YuviPanda: ok to merge? [06:55:59] _joe_: sorry, waiting for pupept to complete on a trusty [06:56:02] _joe_: looks ok on a jessie [06:56:35] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:57:10] _joe_: tools-exec-1409 is not under the tools puppetmaster, btw. Only things that touch k8s is... [06:58:21] <_joe_> YuviPanda: ahah ok cool [06:58:29] <_joe_> I am merging the patch then [06:59:20] _joe_: yup! cool [07:00:09] <_joe_> {{done}} [07:01:02] _joe_: thanks! [07:08:59] _joe_: am off now, thanks for takin care of that [07:13:14] <_joe_> YuviPanda: np, my bad [07:17:43] RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:44] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:22] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:32] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:45] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:59] [13tsreports] 15valhallasw pushed 1 new commit to 06master: 02https://git.io/vr50R [07:30:59] 13tsreports/06master 144dc277d 15Merlijn van Deen: Updated to reflect current situation better [07:31:30] Krinkle: done! As for the redirect... the canonical url was http://stable.toolserver.org/reports/ [07:32:09] but I'm not sure how useful that is [07:33:13] RECOVERY - Puppet run on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:47] RECOVERY - Puppet run on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:06] 06Labs, 10Tool-Labs: jsub appears to act differently towards network requests - https://phabricator.wikimedia.org/T136588#2340420 (10valhallasw) If you use a virtualenv, you have to make sure to use the same ubuntu version on both the host where you built the virtualenv (tools-login, probably, which is trusty)... [07:38:18] RECOVERY - Puppet run on temp-test-trusty-package is OK: OK: Less than 1.00% above the threshold [0.0] [08:27:37] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [09:58:56] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:19] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2340681 (10Aklapper) Is {T136600} a duplicate? SyntaxHighlight_GeSHi also fails for 1.25 (not only 1.26+1.27). [10:19:20] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor, 10SyntaxHighlight: Downloading SyntaxHighlight 1.26 not working -> 404 - https://phabricator.wikimedia.org/T136600#2340687 (10Aklapper) [13:09:30] 06Labs: Reimage labtestmetal2001/labtestweb2001 - https://phabricator.wikimedia.org/T136611#2340919 (10faidon) [13:13:34] !log tools reboot of tools-exec-1203 see T136495 all jobs seem gone now [13:13:35] T136495: Stale NFS handle breaks puppet on tools-exec-1204, -1205 and -1218 - https://phabricator.wikimedia.org/T136495 [13:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [13:37:58] any ETA for https://phabricator.wikimedia.org/T135405 ? :) [13:39:53] Asking for ETAs for tasks is generally hopeless. [13:40:57] ? [13:41:50] . [13:44:34] * Krenair: Do you know where the config is stored, so i can upload a patch myself? [13:45:07] operations/software.git maintain-replicas/maintain-replicas.pl [13:45:13] thx [13:45:23] Good luck getting anyone to run the script, before or after you've modified it. [13:45:42] thx [13:47:05] 06Labs, 10Tool-Labs: Stale NFS handle breaks puppet on tools-exec-1204, -1205 and -1218 - https://phabricator.wikimedia.org/T136495#2336972 (10chasemp) >>! In T136495#2338707, @valhallasw wrote: > This is now also happening on tools-exec-1203: > ``` Thanks @valhallasw ... you rock. So this is a weird thing a... [13:52:51] 06Labs, 10Tool-Labs: Stale NFS handle breaks puppet on tools-exec-1204, -1205 and -1218 - https://phabricator.wikimedia.org/T136495#2340969 (10chasemp) >>! In T136495#2337063, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL, href=https://tools.wmflabs.org/sal/log/AVT5PhKQsSQjJM_mrvwc} [2016-05-28T21:2... [14:12:17] 06Labs, 10Tool-Labs: Stale NFS handle breaks puppet on tools-exec-1204, -1205 and -1218 - https://phabricator.wikimedia.org/T136495#2340986 (10chasemp) Ok went into tools-exec-1218 looking for the race condition and I think I confirmed it. > mount -t nfs > labstore1003.eqiad.wmnet:/dumps on /public/dumps type... [14:25:03] 06Labs, 10Tool-Labs: Stale NFS handle breaks puppet on tools-exec-1204, -1205 and -1218 - https://phabricator.wikimedia.org/T136495#2341024 (10chasemp) p:05Triage>03Normal [14:25:41] 06Labs, 10Tool-Labs: Stale NFS handle breaks puppet on tools-exec-1204, -1205 and -1218 - https://phabricator.wikimedia.org/T136495#2336972 (10chasemp) 05Open>03Resolved a:03chasemp so these hosts in particular are ok now. I'm suspicious of something w/ delayed effect happening here. Let's see if this... [14:36:45] We have been receiving daily email messages from our math instances complaining "Alert: puppet failed on drmf2016.math.eqiad.wmflabs" etc. [14:37:13] Any idea what might be going on here? [14:37:42] Howie: what happens when you run puppet? [14:37:49] 'sudo puppet agent --test' [14:38:59] Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) [14:40:28] This started on 5/20. [14:40:46] Howie: is there still a puppet process running? (ps aux | grep puppet) [14:41:22] no [14:41:58] Howie: in that case, the host probably crashed or was rebooted while puppet was still running. Just remove /var/lib/puppet/state/agent_catalog_run.lock and try running puppet agent -tv [14:45:13] should I put that in the background? [14:46:35] Howie: nope you can run it in teh fg to see output, it just can't run simulatenously to the cron (thus the lock file and sometimes artifact lock file) [14:47:42] ok. i'm running it. Here is the output: http://pastebin.com/EYEJKe2g [14:51:38] 06Labs, 10Tool-Labs: Grid job stuck at 't' state - https://phabricator.wikimedia.org/T136508#2341069 (10chasemp) 05Open>03Resolved a:03chasemp seems none today, I'll close this but reopen if I'm missing something please [14:52:06] So what should we do now? [14:52:54] Howie: htat was partial output I think [14:53:11] need to wait till it errors or finishes [14:53:16] or rereun w/ --debug to see more details [14:53:48] that's all the output so far. [14:54:04] I'll rerun with --debug [14:54:58] ctrl-c doesn't kill it. [14:55:09] 06Labs, 10Tool-Labs: hhvm downgrade breaks puppet on tools-bastion-02 - https://phabricator.wikimedia.org/T136494#2341080 (10chasemp) 05Open>03Resolved a:03chasemp ok now :) apt-get install hhvm -y [14:56:07] ctrl-z doesn't bring prompt either [14:57:48] Howie: that sounds like an nfs issue. Would it be possible to reboot the host? [14:57:56] Howie: can you reboot the isntance via horizon? [14:57:58] ah :) [14:58:37] 06Labs, 10Tool-Labs: puppet disabled on tools-pastion-01 - https://phabricator.wikimedia.org/T136552#2341089 (10chasemp) 05Open>03Resolved a:03chasemp So this is a host I made for sketching out ideas related to T131541. So far the upping of resources along w/ cgroups on tools-bastion-03 seems to be work... [14:59:18] 06Labs, 10Tool-Labs: jsub appears to act differently towards network requests - https://phabricator.wikimedia.org/T136588#2341094 (10chasemp) p:05Triage>03Normal [14:59:22] Can I just do 'sudo shutdown -r now' ? [14:59:37] yeah, that should also work [14:59:52] /sbin/reboot :) [15:01:32] ok I'm doing that now. [15:02:26] ok it's up again [15:03:36] 06Labs, 10Shinken: Describe on http://shinken.wmflabs.org/ what it is about and what credentials are honoured - https://phabricator.wikimedia.org/T88142#2341111 (10chasemp) p:05Triage>03Low [15:04:05] 06Labs: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#2341113 (10chasemp) p:05Triage>03Low [15:04:53] 06Labs: Install package mysql-server on labs debian-jessie prompts 'packages cannot be authenticated' - https://phabricator.wikimedia.org/T122426#2341115 (10chasemp) 05Open>03Resolved a:03chasemp let us know if that didn't fix the issue [15:05:11] 06Labs, 10MediaWiki-Revision-deletion: Need to access revision histories of wikipedia pages - https://phabricator.wikimedia.org/T122035#2341119 (10chasemp) p:05Triage>03Normal [15:05:22] 06Labs, 15User-bd808: Migrate projects using ::role::deprecated::labsvagrant to ::role::labs::mediawiki_vagrant - https://phabricator.wikimedia.org/T121477#2341121 (10chasemp) p:05Triage>03Normal [15:05:43] 06Labs, 06Operations, 10Wikimedia-Video, 07Need-volunteer: Upload the Wikimania 2014 videos to Commons - https://phabricator.wikimedia.org/T106038#2341123 (10chasemp) p:05Triage>03Low [15:07:30] 06Labs, 07LDAP: Restore ldaplist -l passwd - https://phabricator.wikimedia.org/T122595#2341126 (10chasemp) p:05Triage>03Low >>! In T122595#1916607, @MoritzMuehlenhoff wrote: > Ok, I'll simply not use the server control for paged searches when using python-ldap < 2.4, then. can be resolved? [15:07:47] 06Labs: Rebuild jessie image - https://phabricator.wikimedia.org/T122812#2341128 (10chasemp) p:05Triage>03Low [15:07:59] 06Labs, 10Labs-Infrastructure: Setup an apt proxy for labs - https://phabricator.wikimedia.org/T122819#2341129 (10chasemp) p:05Triage>03Normal [15:08:07] 06Labs: labs (labvirt/labservices) in "misc" ganglia cluster - https://phabricator.wikimedia.org/T123000#2341130 (10chasemp) p:05Triage>03High [15:09:22] 06Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#2341134 (10chasemp) p:05Triage>03Normal [15:09:54] 06Labs, 10Tool-Labs: tool-labs error pages HTTP/400 for POSTs - https://phabricator.wikimedia.org/T123136#2341137 (10chasemp) p:05Triage>03Normal [15:10:30] ok it finished this time: here is the output http://pastebin.com/jJdMebqa [15:10:31] 06Labs, 10MediaWiki-extensions-OpenStackManager: OS-EXT-SRV-ATTR:instance_name not set for some instances - https://phabricator.wikimedia.org/T123162#2341140 (10chasemp) p:05Triage>03Normal >>! In T123162#2128231, @Krenair wrote: > Not clear to me whether OSM's expectation that the attribute exists is a re... [15:10:42] 06Labs, 10Tool-Labs: role::labs::tools::proxy tries to create /etc/kubernetes/kubeconfig without requiring /etc/kubernetes - https://phabricator.wikimedia.org/T123176#2341142 (10chasemp) p:05Triage>03Normal [15:12:14] 06Labs, 13Patch-For-Review: Add valhallasw and scfc to labs roots - https://phabricator.wikimedia.org/T123655#2341146 (10chasemp) 05Open>03Resolved a:03chasemp This seems done in relation to @valhallasw, if @scfc is interested I have no objection but I don't want to hold this open indefinitely. [15:12:26] 06Labs, 10MediaWiki-extensions-OpenStackManager: OS-EXT-SRV-ATTR:instance_name not set for some instances - https://phabricator.wikimedia.org/T123162#2341149 (10Krenair) We have other code around depending on it as well, e.g. just from puppet: ```modules/openstack/files/monitor_labs_salt_keys.py: ne... [15:13:27] 06Labs: Access needed to mwui.wmflabs.org - https://phabricator.wikimedia.org/T123316#2341150 (10chasemp) p:05Triage>03Low [15:13:47] RECOVERY - Puppet staleness on tools-pastion-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [15:13:50] 06Labs, 10Tool-Labs, 07Documentation: Document disabling scheduler (#jobs/time) overload protection temporarily - https://phabricator.wikimedia.org/T123411#2341152 (10chasemp) p:05Triage>03Normal [15:14:09] 06Labs, 10Tool-Labs, 13Patch-For-Review: Install libbytes-random-secure-perl on tool labs - https://phabricator.wikimedia.org/T123824#2341153 (10chasemp) 05stalled>03Resolved [15:14:17] 06Labs: Bring back abuse_filter_history view - https://phabricator.wikimedia.org/T123978#2341154 (10chasemp) p:05Triage>03Normal [15:14:47] 06Labs, 10wikitech.wikimedia.org: Semantic search : Provide a search filter for semantic search and a dedicated page to view logged in users' shell access requests. - https://phabricator.wikimedia.org/T124231#2341158 (10chasemp) 05Open>03declined [15:15:10] 06Labs, 10Labs-Infrastructure: Labs bandwidth is aleatory/low - https://phabricator.wikimedia.org/T124960#2341161 (10chasemp) p:05Triage>03Normal [15:15:25] 06Labs: Gussy up Labs proxy 502 page - https://phabricator.wikimedia.org/T125576#2341162 (10chasemp) p:05Triage>03Low [15:15:40] 06Labs, 10BetaFeatures, 10wikitech.wikimedia.org: Enable beta features for Wikitech - https://phabricator.wikimedia.org/T125941#2341164 (10chasemp) p:05Triage>03Normal [15:15:51] 06Labs: Create a "Beginners guide to creating a Labs instance" wiki page - https://phabricator.wikimedia.org/T126094#2341165 (10chasemp) p:05Triage>03Normal [15:16:52] 06Labs, 10Tool-Labs, 10DBA, 06Operations: Replicate wikimania2017wiki to labs - https://phabricator.wikimedia.org/T126096#2341169 (10chasemp) p:05Triage>03Normal [15:17:16] 06Labs, 10Tool-Labs: scripttopic____ uses large amount of memory and swap - https://phabricator.wikimedia.org/T126647#2341171 (10chasemp) 05Open>03Resolved a:03chasemp [15:17:45] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Can not delete images as 'nodepoolmanager' on 'contintcloud' (nodepool account) - https://phabricator.wikimedia.org/T127310#2341176 (10chasemp) 05Open>03Resolved p:05Triage>03Normal a:03chasemp >>! In T127310#2041913, @hashar wrote: > Andrew upload... [15:17:50] 06Labs, 10Labs-Infrastructure: Clean up novaproxy-01 - https://phabricator.wikimedia.org/T136492#2341181 (10AlexMonk-WMF) I removed the old invisible-unicorn package. We should still fix the python version issue. [15:17:55] 06Labs, 10Tool-Labs: Estimate hardware requirements for Tool Labs logging elastic cluster - https://phabricator.wikimedia.org/T127368#2341182 (10chasemp) p:05Triage>03High [15:18:17] 06Labs: Webservice not starting - https://phabricator.wikimedia.org/T127817#2341186 (10chasemp) 05Open>03declined if this is still an issue please add some more detail :) [15:18:28] 06Labs, 10Phlogiston (Technical Debt): phlogiston-2 hangs every week - https://phabricator.wikimedia.org/T129891#2341188 (10chasemp) p:05Triage>03Normal [15:18:33] 06Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#2341190 (10valhallasw) linkwatcher was one of the jobs on the hosts affected by {T136495}; I have resubmitted continuous jobs on the host where it's running now (tools-exec-1205). [15:18:47] 06Labs, 10Tool-Labs: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2341194 (10chasemp) p:05Triage>03Normal [15:19:04] 06Labs, 10Tool-Labs: Labs/Tools mailing list reform - https://phabricator.wikimedia.org/T130637#2341195 (10chasemp) p:05Triage>03Low [15:19:16] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 06Operations, 10hardware-requests: eqiad: (2) Relevance forge servers - https://phabricator.wikimedia.org/T131184#2341198 (10chasemp) p:05Triage>03Normal [15:19:28] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2341199 (10chasemp) p:05Triage>03Normal [15:20:46] chasemp: valhallasw`cloud: Should I do anything else? [15:21:07] Howie: if you want to check if puppet runs correctly now, run puppet agent -tv [15:21:29] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2341204 (10chasemp) 05Open>03declined p:05Triage>03Normal This is a lot of back story my friends. AFAICT this is declined. [15:21:30] (you can also let it be and wait to see if you get another email) [15:21:42] 06Labs, 10PAWS, 10Tool-Labs: Setup a devpi server to help speedup pip installs - https://phabricator.wikimedia.org/T132025#2341208 (10chasemp) p:05Triage>03Low [15:23:31] valhallasw`cloud: ok it finished. Here is output: http://pastebin.com/7frf5p4i [15:23:40] Howie: that looks good! [15:24:11] do you think that solved the problem? just rebooting the instances? [15:24:23] 06Labs: Monitor labs new instance creation - https://phabricator.wikimedia.org/T123590#2341212 (10chasemp) 05Open>03Resolved p:05Triage>03High a:03chasemp [15:24:33] solved-> solves [15:24:35] 06Labs, 10wikitech.wikimedia.org, 07Wikimedia-log-errors: PHP array to string conversion on wikitech in SMW 1.8.x - https://phabricator.wikimedia.org/T124235#2341215 (10chasemp) p:05Triage>03Low [15:25:29] 06Labs, 06Discovery, 06Maps: Maps-warper instance very slow application start up times, passenger timeout - https://phabricator.wikimedia.org/T124538#2341220 (10chasemp) 05Open>03Resolved a:03chasemp >>! In T124538#1959361, @yuvipanda wrote: > Ruby and NFS do not mix well at all, since ruby makes a *lo... [15:25:45] 06Labs, 10Tool-Labs: Add SSHFP dns records to bastions - https://phabricator.wikimedia.org/T132225#2341224 (10chasemp) p:05Triage>03Low [15:26:05] 06Labs: Cleanup proxies that point to nonexistent instances - https://phabricator.wikimedia.org/T132231#2341226 (10chasemp) p:05Triage>03High [15:26:14] 06Labs: Allow to add items to an array of array in InitialiseSettings-labs.php - https://phabricator.wikimedia.org/T132274#2341227 (10chasemp) p:05Triage>03Lowest [15:26:27] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 10MediaWiki-Search, 10wikitech.wikimedia.org: ns and pageid returned from prefixsearch on wikitech are 0 - https://phabricator.wikimedia.org/T132280#2341228 (10chasemp) p:05Triage>03Normal [15:26:38] 06Labs, 10wikitech.wikimedia.org: mwscriptwikiset broken when using all.dblist on terbium - https://phabricator.wikimedia.org/T132383#2341232 (10chasemp) p:05Triage>03Normal [15:27:00] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2341234 (10chasemp) p:05Triage>03Normal @elukey is this still happening? [15:27:17] 06Labs: username case mismatch in keystone totp plugin - https://phabricator.wikimedia.org/T132455#2341239 (10chasemp) p:05Triage>03Normal [15:27:30] 06Labs, 10Wikimedia-General-or-Unknown: [jquery.chosen] Project filter widget missing the 'x' button on the tags in Special:NovaInstance etc - https://phabricator.wikimedia.org/T132480#2341240 (10chasemp) p:05Triage>03Normal [15:28:06] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2341241 (10chasemp) 05Open>03Resolved a:03chasemp [15:28:13] 06Labs: Horizon: Project links on the project panel often drop user into logged-out hell - https://phabricator.wikimedia.org/T133082#2341243 (10chasemp) p:05Triage>03Normal [15:28:37] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2341246 (10chasemp) p:05Triage>03Normal still happening? [15:28:44] 06Labs, 10Tool-Labs: signpostlab and telegrambot webservices flapping (registering/deregistering) - https://phabricator.wikimedia.org/T133092#2341248 (10chasemp) p:05Triage>03High [15:28:53] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Set up (admin-only for now) kubernetes dashboard - https://phabricator.wikimedia.org/T133098#2341249 (10chasemp) p:05Triage>03Normal [15:29:06] 06Labs, 10MediaWiki-Vagrant: mwrepl & hhvmsh do not load wiki in labs vagrant - https://phabricator.wikimedia.org/T133146#2341250 (10chasemp) p:05Triage>03Low [15:29:07] 06Labs, 10wikitech.wikimedia.org: mwscriptwikiset broken when using all.dblist on terbium - https://phabricator.wikimedia.org/T132383#2196393 (10Krenair) what about foreachwikiindblist? [15:29:41] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic: Move californium to an internal host? - https://phabricator.wikimedia.org/T133149#2341254 (10chasemp) p:05Triage>03Normal [15:29:50] 06Labs, 10DBA: archive/archive_userindex is not filled in eswiki_p - https://phabricator.wikimedia.org/T133251#2341255 (10chasemp) p:05Triage>03Normal [15:30:43] 06Labs, 10Tool-Labs, 10DBA: s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133325#2341261 (10chasemp) p:05Triage>03High @merl ping [15:31:32] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2341263 (10chasemp) p:05Triage>03Normal >>! In T133326#2320736, @kaldari wrote: > I deleted all the data older than 2010 as a test. If there are no issues with t... [15:31:40] 06Labs, 10Labs-Infrastructure: Get labs-ns0 and labs-ns1 service IPs in floating space - https://phabricator.wikimedia.org/T133389#2341267 (10chasemp) p:05Triage>03High [15:31:44] 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-Authentication-and-authorization, 06Reading-Infrastructure-Team: Update OpenStackManager to use AuthManager - https://phabricator.wikimedia.org/T110288#2341270 (10Anomie) [15:32:11] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10Monitoring, 06Operations: Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656#2341273 (10chasemp) p:05Triage>03High I believe this is still happening on infrequently [15:33:42] 06Labs, 10Shinken: Shinken timeouts - https://phabricator.wikimedia.org/T134024#2341281 (10chasemp) p:05Triage>03Low I think this is related to this host being overloaded / slow [15:34:45] 06Labs, 10Shinken: Shinken timeouts - https://phabricator.wikimedia.org/T134024#2341289 (10Krenair) Yes, it might be {T127957} [15:35:24] 06Labs: Monitor labs new instance creation - https://phabricator.wikimedia.org/T123590#2341292 (10Andrew) 05Resolved>03Open [15:35:57] 06Labs: dumps-stats.dumps.eqiad.wmflabs instance was hammering NFS - https://phabricator.wikimedia.org/T134148#2341296 (10chasemp) p:05Triage>03Normal @hydriz did you change your behavior here at all? Seems to have been less of an issue [15:36:01] 06Labs, 10DBA, 10MediaWiki-General-or-Unknown: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2341298 (10chasemp) p:05Triage>03Normal [15:36:14] 06Labs, 10Tool-Labs: Update nginx on tools and labs proxies and static file server - https://phabricator.wikimedia.org/T134383#2341301 (10chasemp) p:05Triage>03Normal [15:36:22] 06Labs, 10Tool-Labs, 07Tracking: Simplify and reduce the amount of options jsub supports (Tracking) - https://phabricator.wikimedia.org/T134846#2341302 (10chasemp) p:05Triage>03Normal [15:36:24] 06Labs: username case mismatch in keystone totp plugin - https://phabricator.wikimedia.org/T132455#2341303 (10Andrew) It's possible that this was an incidence of https://phabricator.wikimedia.org/T131630 -- coren, care to try again? [15:38:14] 06Labs, 10DBA, 10MediaWiki-General-or-Unknown: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2341307 (10jcrespo) 05Open>03Invalid As no one contradicted me, and even more people supported my thesis (we do not use user_editcount), I will mark this as inva... [15:38:21] 06Labs: More local storage on a wmflabs vm? - https://phabricator.wikimedia.org/T134986#2341311 (10chasemp) 05Open>03declined p:05Triage>03Normal >>! In T134986#2304384, @Gehaxelt wrote: > Bump? > @Physikerwelt Thanks for checking this. > > @Andrew It would be nice if you could increase the quota for t... [15:38:51] 06Labs: Backup files request - https://phabricator.wikimedia.org/T135014#2341316 (10chasemp) p:05Triage>03Normal [15:39:09] 06Labs, 10Mail: failed exim service on labs instances - https://phabricator.wikimedia.org/T135033#2341317 (10chasemp) p:05Triage>03Normal [15:39:27] 06Labs, 10Labs-Infrastructure, 06WMF-Legal, 07Privacy: Whitelist labs instances that need XFF header passed through the web proxy - https://phabricator.wikimedia.org/T135046#2341318 (10chasemp) p:05Triage>03High [15:39:44] 06Labs, 10Labs-Infrastructure, 10Quarry: Long-running Quarry query (querry?) produces strangely incorrect results - https://phabricator.wikimedia.org/T135087#2341321 (10chasemp) p:05Triage>03Normal @yuvipanda any ideas? [15:39:55] 06Labs, 10Tool-Labs: jsub's -once should clear jobs in E state and run things - https://phabricator.wikimedia.org/T135229#2341323 (10chasemp) p:05Triage>03Normal [15:40:03] 06Labs, 10DBA: Replicate CN related tables at labs - https://phabricator.wikimedia.org/T135405#2341324 (10chasemp) p:05Triage>03Normal [15:40:09] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2341327 (10chasemp) p:05Triage>03Normal [15:40:15] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2341328 (10chasemp) p:05Triage>03High [15:40:49] 06Labs, 10Labs-Infrastructure, 10DBA: labsdb1001 crashed yesterday at 21:48:07 - https://phabricator.wikimedia.org/T135971#2341331 (10chasemp) p:05Triage>03High @jcrespo anything we can do to help? [15:41:10] 06Labs, 10wikitech.wikimedia.org, 07Regression: Wikitech sign-up page has bad styling - https://phabricator.wikimedia.org/T136032#2341333 (10chasemp) p:05Triage>03Low [15:42:12] 06Labs: novaproxy 502's due to intermittent DNS failures - https://phabricator.wikimedia.org/T136073#2341335 (10chasemp) 05Open>03Resolved a:03chasemp We were in the process of reimaging the secondary DNS server which seemed to cause issues for these proxies in that they were hammering DNS and dropping req... [15:42:24] 06Labs, 10Tool-Labs: templatetiger is using 613G in Tools out of 8T - https://phabricator.wikimedia.org/T136192#2341339 (10chasemp) p:05Triage>03Normal [15:42:30] 06Labs, 10DBA, 07Tracking: Labs users missing grants on replicas (tracking) - https://phabricator.wikimedia.org/T136319#2341340 (10chasemp) p:05Triage>03Normal [15:42:44] 06Labs, 10MediaWiki-extensions-OATHAuth: Move two-factor auth data (TOTP seed) from labswiki database to LDAP - https://phabricator.wikimedia.org/T136350#2341341 (10chasemp) p:05Triage>03Normal [15:43:01] valhallasw`cloud: any idea why? https://phabricator.wikimedia.org/T136404 [15:43:18] 06Labs, 10Tool-Labs: Queues disabled on tools-exec-1407.eqiad.wmflabs, tools-exec-1216.eqiad.wmflabs, tools-exec-1219.eqiad.wmflabs - https://phabricator.wikimedia.org/T136404#2341343 (10chasemp) p:05Triage>03High any clue as to why? [15:43:32] 06Labs, 10Labs-Infrastructure: Clean up novaproxy-01 - https://phabricator.wikimedia.org/T136492#2341345 (10chasemp) p:05Triage>03Normal [15:43:42] 06Labs, 10Tool-Labs: install php5-readline on bastion and exec hosts - https://phabricator.wikimedia.org/T136519#2341346 (10chasemp) p:05Triage>03Low [15:44:18] 06Labs, 10MediaWiki-extensions-OpenStackManager: OS-EXT-SRV-ATTR:instance_name not set for some instances - https://phabricator.wikimedia.org/T123162#2341348 (10Andrew) I think that value was the ec2 id -- it doesn't appear to be reported by nova anymore and, in any case, I'd prefer we not use it anywhere. Th... [15:44:50] chasemp: no, I created the bug because I didn't know :-) [15:44:56] 06Labs, 10DBA, 07Tracking: Labs users missing grants on replicas (tracking) - https://phabricator.wikimedia.org/T136319#2341357 (10jcrespo) [15:44:57] probably because they had to be rebooted [15:44:58] 06Labs, 10Tool-Labs: labsdb accounts being created without grants to create personal databases - https://phabricator.wikimedia.org/T130595#2341352 (10jcrespo) 05Open>03Resolved a:03jcrespo Fixed on T135947 AFAIK. [15:45:01] or moved between virt hosts, etc [15:45:45] 06Labs, 10DBA, 07Tracking: Labs users missing grants on replicas (tracking) - https://phabricator.wikimedia.org/T136319#2330501 (10jcrespo) 05Open>03Resolved a:03jcrespo Fixed on T135947, no reason to track unless it happens again. [15:45:54] I'm also not sure how to figure out when they were disabled [15:46:09] 06Labs, 10Labs-Infrastructure, 10DBA: labsdb1001 crashed yesterday at 21:48:07 - https://phabricator.wikimedia.org/T135971#2341370 (10jcrespo) 05Open>03Resolved a:03jcrespo Al immediate actionables were already done. [15:46:12] ok thanks, and same [15:47:13] hm, maybe in messages? /me checks [15:57:37] 06Labs, 10Tool-Labs: Queues disabled on tools-exec-1407.eqiad.wmflabs, tools-exec-1216.eqiad.wmflabs, tools-exec-1219.eqiad.wmflabs - https://phabricator.wikimedia.org/T136404#2341420 (10valhallasw) FWIW, these queues seem to have been disabled may 20th around 2100 UTC (-1407), 1550 UTC (-1216) and 2120 UTC (-... [15:57:39] chasemp: ^ [15:57:58] shall I reenable them? [16:00:01] 06Labs, 10Tool-Labs: tools.suggestbot web requests fail after a period of time - https://phabricator.wikimedia.org/T133090#2341424 (10Nettrom) With regards to the web service being restarted: I count 80 "No running webservice" notifications in `~suggestbot/service.log` for 2016-05-30. There are a handful of re... [16:07:35] 06Labs, 10DBA: Wrong page title in labs database replica enwiki page table - https://phabricator.wikimedia.org/T136618#2341449 (10Bamyers99) [17:26:46] valhallasw`cloud: assuming hosts in queues :) yeah I say go for it [17:32:51] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace.root.byte_percentfree (<10.00%) [17:33:03] YuviPanda: ^ [17:35:26] !log tools re-enabled queues on tools-exec-1407, tools-exec-1216, tools-exec-1219 [17:35:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [17:35:42] 06Labs, 10Tool-Labs: Queues disabled on tools-exec-1407.eqiad.wmflabs, tools-exec-1216.eqiad.wmflabs, tools-exec-1219.eqiad.wmflabs - https://phabricator.wikimedia.org/T136404#2341748 (10valhallasw) 05Open>03Resolved a:03valhallasw Queue re-enabled. [17:36:25] All queues on all hosts are OK now! :-) [17:37:52] it's a christmas miracle [17:48:29] 06Labs, 06Discovery, 06Discovery-Search-Backlog, 10MediaWiki-Search, 10wikitech.wikimedia.org: ns and pageid returned from prefixsearch on wikitech are 0 - https://phabricator.wikimedia.org/T132280#2341848 (10Deskana) 05Open>03Resolved a:03Deskana The query given in the description now gives zero r... [17:54:00] 06Labs, 10Tool-Labs: wikiviewstats is using 232G on Tools - https://phabricator.wikimedia.org/T136198#2341857 (10chasemp) >>! In T136198#2337465, @Cyberpower678 wrote: > I'm not sure why this is assigned to me. I'm not too familiar with wikiviewstats. I can't find this massive directory as mentioned "bak" do... [18:00:27] 06Labs, 07LDAP: Restore ldaplist -l passwd - https://phabricator.wikimedia.org/T122595#1908530 (10scfc) I am interested in @MoritzMuehlenhoff's patch. [18:02:45] 06Labs, 10Tool-Labs: wikiviewstats is using 232G on Tools - https://phabricator.wikimedia.org/T136198#2341898 (10Betacommand) Not sure how I got listed as a maintainer, I am not involved with this project. [18:17:16] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2341950 (10chasemp) a:05chasemp>03Krenair @krenair can you elaborate on who this indicates: > then the task is not actually fixed until that includes the people who are suppo... [18:17:47] 06Labs: Monitor labs new instance creation - https://phabricator.wikimedia.org/T123590#2341955 (10chasemp) a:05chasemp>03None thanks @andrew I did not intend to close this prematurely [18:22:00] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor, 10SyntaxHighlight: Downloading SyntaxHighlight 1.26 not working -> 404 - https://phabricator.wikimedia.org/T136600#2341958 (10Paladox) It seems to be linking to https://extdist.wmflabs.org/dist/extensions/SyntaxHighlight_GeSHi-REL1_26-aa21d... [18:35:28] Does anyone here have an Android phone with Google Authenticator? [18:36:05] Wondered if there was any way to backup your Wikitech 2fa details. Just formatted my phone and now I've lost access to my Wikitech account [18:36:44] I can prove ownership of my account via SSH, so would love some help by a Labs administrator. [18:37:42] SPF|Cloud: hey [18:38:05] hi [18:38:06] SPF|Cloud: do you have access to phabricator? [18:38:10] yes [18:38:14] SPF|Cloud: if so can you open a phab ticket and I'll help you now [18:39:52] https://phabricator.wikimedia.org/T136634 [18:40:50] 06Labs: Lost Wikitech 2FA details, recovery needed - https://phabricator.wikimedia.org/T136634#2342032 (10yuvipanda) [18:41:20] 06Labs: Lost Wikitech 2FA details, recovery needed - https://phabricator.wikimedia.org/T136634#2342021 (10yuvipanda) can you write the following string into a file in your homedir on tools, and tell me the name of the file? ``` oht0ipe1Pho7aa7pohChie8eath0ogoo9Eesoh9nahc3aefoh7ie2ais6oohugoo ``` Thanks. [18:42:51] heh bah, tools-exec.wikimedia.org or tools-exec.tools.eqiad.wmflabs doesn't work? [18:43:06] what are you tryng to do? [18:43:21] I'm currently logged in at the Labs bastion and try to SSH into a tools host [18:45:46] Is tools-exec-1407 good enough? [18:46:18] SPF|Cloud: all tool hosts are on nfs, so sure. You could also login directly to tools-login.wmflabs.org [18:46:21] SPF|Cloud: they all have same nfs, so sure :) you can also just login directly to login.tools.wmflabs.org without having to go through a bastion too [18:46:23] hah [18:46:34] Oh, addresses changed :p [18:46:38] 06Labs: Lost Wikitech 2FA details, recovery needed - https://phabricator.wikimedia.org/T136634#2342037 (10Southparkfan) ``` southparkfan@tools-exec-1407:~$ echo 'oht0ipe1Pho7aa7pohChie8eath0ogoo9Eesoh9nahc3aefoh7ie2ais6oohugoo' > reset_2fa.txt southparkfan@tools-exec-1407:~$ cat reset_2fa.txt oht0ipe1Pho7aa7pohC... [18:46:52] ^ there you go [18:49:53] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2339090 (10mmodell) I don't see any files with recent time stamps in https://extdist.wmflabs.org/dist/extensions/ - the newest is `03-May-2016 07:00` [18:50:04] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2339090 (10Paladox) It happends for CentralAuth too. Seems the files aren't new either some showing as from January when we cut the branch only a few m... [18:50:50] SPF|Cloud: kk gimme a bit [18:52:23] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2342059 (10mmodell) @luke081515 I'll work on it a bit and see if I can get it to be more automated. [18:52:31] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2342060 (10mmodell) a:03mmodell [18:55:54] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2342072 (10Paladox) It seems skins works just extensions doint. [18:58:54] 06Labs: Lost Wikitech 2FA details, recovery needed - https://phabricator.wikimedia.org/T136634#2342101 (10Andrew) 05Open>03Resolved a:03Andrew Done. [18:58:56] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2342104 (10Legoktm) It appears that `/srv` is full: /dev/mapper/vd-second--local--disk 21G 20G 0 100% /srv [19:02:26] Hey is there any bandwidth throttling on labs servers web proxies? [19:02:38] (03PS1) 10Merlijn van Deen: Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291965 (https://phabricator.wikimedia.org/T136623) [19:02:51] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: connect usb external disk to labmon1001 - https://phabricator.wikimedia.org/T136242#2342154 (10Cmjohnson) Connected a 3TB disk with the usb drive toaster. Did not mount. [19:02:56] (03CR) 10jenkins-bot: [V: 04-1] Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291965 (https://phabricator.wikimedia.org/T136623) (owner: 10Merlijn van Deen) [19:03:02] brion: not that I'm aware of [19:03:06] I'm testing some alternate transcoding settings and sometimes see slow/stalled downloads from media-streaming.wmflabs.org (the index is not yet pretty, beware) [19:03:09] might be on my end :) [19:03:14] * brion shakes fist at comcast [19:03:19] brion: except for the situation 'the proxy is overloaded' [19:03:31] currently i'm seeing about 75-80mbits which ain't bad [19:04:21] brion: there probably are settings we could tweak in the nginx wrt buffering or something [19:04:25] sometimes drops lower though [19:04:30] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: connect usb external disk to labmon1001 - https://phabricator.wikimedia.org/T136242#2342163 (10Cmjohnson) a:05Cmjohnson>03RobH [19:04:57] YuviPanda: nah if it's just sometimes overloading cause i'm a bw hog then there's probably not much to tweak [19:05:05] :D ok! [19:05:29] just checking to make sure i'm not using up a quota ;) [19:05:53] :D [19:05:55] ok [19:06:17] oh hey -- i noticed it's easy to create an instance that i can't log in to if i forget to attach my ssh public key in the appropriate tab on horizon [19:06:30] am i forgetting to set a default or should i just remember to set it every time? [19:06:40] andrewbogott: ^ [19:06:59] (03PS1) 10Merlijn van Deen: Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) [19:07:18] brion: If the ssh key/horizon gui is visible and enabled that's a mistake [19:07:30] everything should be handled via your project membership and your keys in ldap [19:07:32] ok lemme pull it up again and double-check [19:07:33] (03CR) 10jenkins-bot: [V: 04-1] Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) (owner: 10Merlijn van Deen) [19:07:51] (03Abandoned) 10Merlijn van Deen: Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291965 (https://phabricator.wikimedia.org/T136623) (owner: 10Merlijn van Deen) [19:08:04] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2342193 (10Paladox) >>! In T136564#2342104, @Legoktm wrote: > It appears that `/srv` is full: > > /dev/mapper/vd-second--local--disk 21G 20G 0... [19:08:04] * valhallasw`cloud curses [19:09:25] brion: ah, I remember now, I can't easily disable that misleading ssh gui without also disabling the security groups [19:09:35] it's more modular in future versions :( [19:09:40] aha [19:10:00] I guess maybe that gui is a bonus feature… but mostly things are done via project membership [19:10:18] andrewbogott: so what happened to me the other day was i created a debian instance in ogvjs-integration, *didn't* fill anything out in the 'access & security' tab, and was unable to log in to the resulting instance [19:10:37] does that instance still exist? [19:10:39] i terminated the instance and recreated it, adding my ssh key in on the 'access & security' tab, and that instance works [19:10:43] alas no i killed the first one :( [19:10:44] oh [19:10:47] no forensics available [19:10:49] sorry! [19:10:51] Do you have a working key registered with wikitech? [19:11:00] (03PS2) 10Merlijn van Deen: Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) [19:11:11] lemme double check [19:11:32] (03CR) 10jenkins-bot: [V: 04-1] Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) (owner: 10Merlijn van Deen) [19:12:20] (03PS3) 10Merlijn van Deen: Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) [19:12:45] Oh, fun. Python 2.7 and 3.4 disagreeing on the exception format [19:12:56] (03CR) 10jenkins-bot: [V: 04-1] Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) (owner: 10Merlijn van Deen) [19:13:34] andrewbogott: ... looks like i have one current key but it might not have been the one i have explicitly selected for labs [19:13:47] so it's possible it was using the old one from wikitech by default? maybe? [19:14:18] hmmmm, no i think ssh was trying all my keys though [19:14:22] damn i shoulda saved those logs. sorry! [19:14:26] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2342234 (10Legoktm) I deleted some extra tarballs that aren't needed anymore. The root cause of this appears to be {T123180} - we're keeping around tarb... [19:14:44] brion: I think it's very likely that anything you selected in that horizon gui had, at best, a placebo effect. [19:14:50] haha [19:14:52] Because it would most likely have been set up as a root key in any case. [19:14:53] very possible yes [19:15:12] (I don't know for sure what that feature does but I think it's strictly root keys) [19:15:15] if i encounter it again i'll save the broken instance! [19:15:20] ok :) [19:15:24] sorry I don't have a better answer [19:15:29] no worries [19:21:24] (03PS4) 10Merlijn van Deen: Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) [19:23:01] (03CR) 10Merlijn van Deen: [C: 032] Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) (owner: 10Merlijn van Deen) [19:23:32] (03Merged) 10jenkins-bot: Make Bug: parsing more lenient [labs/tools/forrestbot] - 10https://gerrit.wikimedia.org/r/291966 (https://phabricator.wikimedia.org/T136623) (owner: 10Merlijn van Deen) [19:44:27] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2342402 (10Paladox) @legoktm thanks for fixing the problem. Its now working. MobileFrontend and CentralAuth work now. [19:46:04] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor, 10SyntaxHighlight: Downloading SyntaxHighlight 1.26 not working -> 404 - https://phabricator.wikimedia.org/T136600#2342406 (10Paladox) 05Open>03Resolved a:03Paladox @Legoktm found the storage was full and needed some files removed. A... [19:46:20] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor, 10SyntaxHighlight: Downloading SyntaxHighlight 1.26 not working -> 404 - https://phabricator.wikimedia.org/T136600#2342410 (10Paladox) a:05Paladox>03Legoktm [20:15:35] (03PS2) 10BryanDavis: www: cleanup minor index.php issues [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291072 [20:15:45] (03CR) 10BryanDavis: [C: 032] www: cleanup minor index.php issues [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291072 (owner: 10BryanDavis) [20:16:28] (03Merged) 10jenkins-bot: www: cleanup minor index.php issues [labs/toollabs] - 10https://gerrit.wikimedia.org/r/291072 (owner: 10BryanDavis) [21:04:32] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2342696 (10Qgil) I wonder whether Mr.Z-bot would be easier to maintain in combination of https://tools.wmflabs.org/massviews/ or whetever APIs that tools is querying... [21:12:51] 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2342722 (10russblau) Is there any update on the status of this? On 23 May, the revision table was in progress and was expected to take ~12 hours. The pagelinks table is about 3X larger and so might be expected... [21:22:37] (03PS1) 10Legoktm: Don't overload the `branch` variable [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/292032 (https://phabricator.wikimedia.org/T123180) [21:27:21] (03CR) 10Paladox: [C: 031] Don't overload the `branch` variable [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/292032 (https://phabricator.wikimedia.org/T123180) (owner: 10Legoktm) [21:27:56] (03CR) 10Legoktm: [C: 032] Don't overload the `branch` variable [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/292032 (https://phabricator.wikimedia.org/T123180) (owner: 10Legoktm) [21:28:25] (03Merged) 10jenkins-bot: Don't overload the `branch` variable [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/292032 (https://phabricator.wikimedia.org/T123180) (owner: 10Legoktm) [21:29:58] 10Labs-project-extdist: Create new extdist instance - https://phabricator.wikimedia.org/T88787#2342781 (10Legoktm) 05Open>03Resolved a:03yuvipanda Yuvi did this a long time ago. [21:30:13] 10Labs-project-extdist, 10MediaWiki-extensions-ExtensionDistributor: Download snapshot generates 404 for downloads - https://phabricator.wikimedia.org/T136564#2342791 (10Legoktm) 05Open>03Resolved [21:34:29] 10Labs-project-extdist, 13Patch-For-Review: extdist is not deleting some tarballs for master - https://phabricator.wikimedia.org/T123180#2342816 (10Legoktm) 05Open>03Resolved a:03Legoktm [21:36:48] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2342827 (10kaldari) @Qgil: As soon as T118508 is fixed, it should actually be pretty easy to replace this tool entirely. Unfortunately, due to T118508, https://tools... [21:53:58] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2342903 (10kaldari) @Mr.Z-man: Hmm, it looks like the tool hasn't actually run since April. Any idea what might be wrong with it? What's the command to run it manual... [22:17:49] 06Labs, 10Tool-Labs, 10DBA: p50380g50816__pop_stats (popularpages) using 53G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133326#2343009 (10Mr.Z-man) I've actually been running it manually for a while, because it has never been quite as reliable as I was hoping. For some reason in April it mis... [22:26:43] YuviPanda: tested on self hosted instance and seems to work fine - https://gerrit.wikimedia.org/r/#/c/292030/ [22:27:15] madhuvishy: can you make patches for all the uses of uwsgi::app too? [22:27:20] so we have one consistent way of doing this? [22:27:29] I can babysit them all too [22:27:41] YuviPanda: hmmm passing in plugin is optional though [22:27:53] i can do it - but they can give it inside settings or outside [22:27:55] madhuvishy: but it doesn't really work without passing in plugin [22:28:00] outside would force override [22:28:02] it does [22:28:07] madhuvishy: what does it load? [22:28:10] just assumes python3 [22:28:12] if you don't pass in plugin? [22:28:15] oh [22:28:17] lol [22:28:54] that also for me - because i was giving callable - it went - ooh callable - ooh i found it in the python3 plugin - sure i'll use that [22:29:01] hmm [22:29:16] madhuvishy: so I think there should only be one recommended way to specify plugin [22:29:28] and people shouldn't have to know which ones (like callable) will override [22:29:29] yeah that makes sense [22:29:44] you can also do plugin or plugins [22:29:50] both take lists btw [22:29:57] No difference :/ [22:30:14] but yes, i can patch the modules [22:30:46] madhuvishy: yeah, let's just use plugins. I also found another problem, commented [22:31:22] madhuvishy: am gonna brb, need to wash hair [22:31:31] ya okay [23:47:09] Hi everyone! :) [23:49:19] yoyo [23:50:40] I have a problem using the shell of Pywikibot As Web Service. [23:51:12] hello Ivanhercaz [23:51:18] what are you running into? [23:51:46] I'm a Spanish user and I need to write accents, but when I press an accent it isn't writed [23:51:57] Hi YuviPanda [23:52:45] yes unfortunately that is a known issue [23:52:48] a workaround right now [23:52:53] is to create a new text file in a tab [23:52:56] and write your commands there [23:53:04] and execute the file with 'bash ' in the terminal [23:53:40] oh, okay. [23:53:54] 10PAWS: PAWS public will not allow for downloading a whole TSV file - https://phabricator.wikimedia.org/T130132#2343294 (10yuvipanda) 05Open>03Resolved a:03yuvipanda This has been fixed now, since we just serve it with nginx. [23:54:05] My problem is when I write the name of a file that I want to search to replace [23:54:15] 10PAWS: Split proxy from hub in PAWS - https://phabricator.wikimedia.org/T129208#2343297 (10yuvipanda) 05Open>03Resolved a:03yuvipanda This is done too. I've also switched to the nginx proxy. [23:54:26] Ivanhercaz: so what is the command you want to write? [23:54:46] https://phabricator.wikimedia.org/T136118 is the bug, btw [23:56:25] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 10grrrit-wm: Fix grrrit-wm access situation - https://phabricator.wikimedia.org/T132828#2343303 (10Krenair) a:05Krenair>03chasemp >>! In T132828#2341950, @chasemp wrote: > @krenair can you elaborate on who this indicates: > >> then the task is not actually fixed... [23:56:32] I execute a bash file, then I write the file that I want to search and it writes an python replace.py -filelinks:"$pageGen" [23:56:46] $pageGen is the name of the file [23:57:24] Any alternative way to write an accent? [23:58:26] Ivanhercaz: you can write that command in a text file (new -> 'text file' in paws after you log in), and save it with a name (you can change name by clicking on the 'Untitled' on top) [23:58:34] and then on terminal you can just write 'bash ' [23:58:39] 10Labs-project-wikistats, 13Patch-For-Review: delete orain and pardus tables from wikistats - https://phabricator.wikimedia.org/T136460#2343307 (10Dzahn) 05Open>03Resolved done, merged and deployed wikistats with deploy script dropped tables from db [23:58:46] this is exactly equivalent to typing the command in the terminal [23:59:03] and in 'new -> text file' you can type accents as you normally would [23:59:05] and it will work