[00:25:05] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [00:26:27] (03CR) 10Tim Landscheidt: "recheck" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/268934 (owner: 10Tim Landscheidt) [00:36:05] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [00:41:03] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [00:42:24] hello, is this the place to mention that discourse is down? https://discourse.wmflabs.org [00:50:51] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2006998 (10scfc) The triggering package has now moved to `groff-base` and others (https://integration.wikimedia.org/ci/job/debian-glue/89/... [01:01:20] samwilson, best option would be to contact the person responsible for maintaining it [01:01:38] labs projects are not really actively managed centrally [01:06:12] @krenair, thanks, will do :) [01:08:12] (03PS1) 10Legoktm: jenkins job validation, do not submit [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/269074 [01:12:40] hi [01:13:04] (03Abandoned) 10Legoktm: jenkins job validation, do not submit [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/269074 (owner: 10Legoktm) [01:18:16] hi [01:21:03] 10Labs-Other-Projects: Succesful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#2007048 (10Samwilson) https://discourse.wmflabs.org/ is currently down: 502 Bad Gateway [03:01:06] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [03:11:05] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [03:16:42] anyone here? [03:16:42] Hi Peter____, I am here, if you need anything, please ask, otherwise no one is going to help you... Thank you [03:17:09] Talking to a bot is not my favorite... [03:17:25] ;) [03:22:04] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [03:26:42] ich hab doch nur ne frage ... [03:53:52] twentyafterfour: I am in phab-02. did we have a problem with that one? phab-01 still broken though [03:54:16] Negative24: I wasn't able to log in to either one before [03:54:20] and phab-03 doesn't have a signed puppet cert... [03:54:30] hmm [03:54:35] Negative24: still can't log in to phab-02 either [03:54:42] interesting [03:55:12] Negative24: can you look at the logs, say /var/log/syslog, and look for anything interesting? [03:55:19] sure [03:55:25] puppets broken on -02 [03:55:34] probably misconfigured address [03:56:18] /tmp is full [03:57:51] So, we have a 1MiB /tmp partition and its full from a mkinitramfs. Probably from a kernel upgrade [04:02:17] "User twentyafterfour from bastion-01.bastion.eqiad.wmflabs not allowed because not listed in AllowUsers" [04:02:22] twentyafterfour: ^ [04:07:25] twentyafterfour: try logging into phab-02 again [04:22:05] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [04:28:04] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [04:58:05] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [05:04:03] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [05:39:06] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [06:10:44] !log set $wgAuthenticationTokenVersion on beta cluster (test run for T124440) [06:10:45] set is not a valid project. [06:10:54] !log deployment-prep set $wgAuthenticationTokenVersion on beta cluster (test run for T124440) [06:10:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [06:12:41] We should figure out how to make !log to deployment-prep not work [06:29:24] what does an instance state of shutoff mean? or more accurately, how can i turn it back on :) [06:30:52] ebernhardson: I assume there is no "boot" or "reboot" action in the action column? [06:31:08] bd808: i tried reboot, but to no avail :( [06:33:37] poked through the logs on wikitech, most mentions seem to indicate it means something went wrong in the underlying machinery :S last console output was on jan 24 so no help there either .. oh well i'll poke some people in the morning i guess [06:34:12] ebernhardson: I'd open a phab ticket. ch.ase or others may start their day before you do [06:35:15] ebernhardson: does https://wikitech.wikimedia.org/wiki/MediaWiki:Cirrussearch-boost-templates take effect magically? [06:35:33] Or does something have to happen to reindex and change the boost? [06:37:45] 6Labs: Instance discourse.search.eqiad.wmflabs in SHUTDOWN state - https://phabricator.wikimedia.org/T126191#2007200 (10EBernhardson) 3NEW [06:38:11] bd808: it gets parsed and cached into memcached, i don't think for particularly long though. lemme check [06:38:42] ten minutes [06:38:48] is it not working? [06:38:49] hmm [06:39:03] there is also a config flag that has to be enabled for them to be used, lemme check if its on for wikitech [06:39:13] (or there is a url param to override the config flag) [06:41:05] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [06:41:07] This search is still showing Nova Resource pages pretty high -- https://wikitech.wikimedia.org/w/index.php?search=mediawiki&title=Special%3ASearch&go=Go [06:42:01] but maybe I just didn't de-boost enough [06:43:33] hmm, yea it doesn't look to be enabled for wikitech based on that query. You can append &cirrusDumpQuery to any search to get the query we send to es. [06:43:58] you can see at the bottom of https://en.wikipedia.org/w/index.php?search=~usa&title=Special%3ASearch&go=Go&cirrusDumpQuery it has a bunch of weighted filters in the rescore_query [06:44:14] the code is turning out harder to dig through than i remember :P [06:45:40] ebernhardson: if you find something that needs configured, could you leave a note on https://phabricator.wikimedia.org/T122993 ? [06:45:44] * bd808 needs to sleep [06:45:58] sure [06:46:03] thx [06:56:04] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [07:17:22] turns out the regex doesn't accept negative numbers. Woo parsing with regular expressions :P [07:18:45] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2007217 (10EBernhardson) The regular expression that reads these doesn't like negative numbers. Just use a low %, like 10% or something. [08:49:54] (03PS1) 10Legoktm: Add README, re-license as GPL [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/269089 [08:51:30] (03CR) 10Legoktm: [C: 032] Add README, re-license as GPL [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/269089 (owner: 10Legoktm) [08:52:09] (03Merged) 10jenkins-bot: Add README, re-license as GPL [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/269089 (owner: 10Legoktm) [08:59:05] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [10:12:55] 6Labs, 10Tool-Labs: tools-exec-12* puppet broken: php5* packages have been upgraded again? - https://phabricator.wikimedia.org/T126205#2007397 (10valhallasw) 3NEW [10:14:06] RECOVERY - SSH on tools-webgrid-lighttpd-1208 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [10:20:06] PROBLEM - SSH on tools-webgrid-lighttpd-1208 is CRITICAL: Server answer [10:36:03] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2007454 (10mforns) @MusikAnimal >> My fork of marcelrf's app just adds history push states so it can support deep linking (the pop states I didn't finish... b... [10:47:03] (03CR) 10Hashar: "recheck" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/267632 (owner: 10Tim Landscheidt) [10:47:30] (03CR) 10Hashar: "This change passed debian-glue just fine. Rechecking to investigate T125999" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/267632 (owner: 10Tim Landscheidt) [10:48:50] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2007473 (10hashar) I am trying to update the cow image manually with: ``` jenkins-deploy@integration-slave-jessie-100... [11:14:47] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2007509 (10hashar) I tried tweaking the $basepath in puppet, but that is not the issue actually though we should stil... [12:11:49] PROBLEM - SSH on tools-worker-1002 is CRITICAL: Server answer [12:50:59] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2007646 (10hashar) The reason for the symlink of sid/unstable is {T111097} [12:54:55] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7WorkType-Maintenance: Change sid pbuilder image name to 'unstable' - https://phabricator.wikimedia.org/T111097#2007656 (10hashar) Funny side effect found on {T125999}. The labs/toollabs repo mentions `unstable` and thus the... [12:57:33] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng, 5Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2007660 (10hashar) Something I don't quite understand yet is that the `base-unstable-amd64.cow` is a symlink to `base... [13:31:56] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2007685 (10TheDJ) @MusikAnimal btw. It seems that there is code to strip www. from entry, but www.wikidata.org is the official url for wikidata, so it's currentl... [13:33:19] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2007686 (10Sjoerddebruin) Also @MusikAnimal, your tool doesn't seems to work in Safari here. The input field to select pages doesn't function. [13:35:29] RECOVERY - Puppet staleness on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:55:18] (03PS1) 10Tim Landscheidt: WIP [labs/toollabs] - 10https://gerrit.wikimedia.org/r/269132 [14:55:45] (03CR) 10jenkins-bot: [V: 04-1] WIP [labs/toollabs] - 10https://gerrit.wikimedia.org/r/269132 (owner: 10Tim Landscheidt) [14:58:41] I'd like to test a puppet change on labs (https://phabricator.wikimedia.org/T109101). Where can I find how to use the lab's puppetmaster? [14:59:28] gehel: https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [15:00:08] this documents the step to setup/use your own puppet master in labs [15:00:33] moritzm: seems to be exactly what I was looking for. Thanks ! [15:06:12] is the puppet compiler labs instance dead? [15:07:33] nevermind, it's back! [15:09:55] PROBLEM - Puppet failure on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:11:58] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:12:40] (03PS2) 10Tim Landscheidt: WIP [labs/toollabs] - 10https://gerrit.wikimedia.org/r/269132 [15:13:06] (03CR) 10jenkins-bot: [V: 04-1] WIP [labs/toollabs] - 10https://gerrit.wikimedia.org/r/269132 (owner: 10Tim Landscheidt) [15:13:27] nfs probs? [15:14:11] dns issues [15:14:20] it keeps happening often for the labservices hosts [15:15:17] dns issues because the dns->ldap thing fails because ldap is intermittent, at least from the dns servers? I know we saw that before in the distant past [15:16:27] I'm not sure of the why bblack but a restart of pdns & pdns-recursor seems to clear it up on labservices1001.wikimedia.org when I have seen it (second time now) [15:16:35] but andrewbogot.t has said he has seen it happening more often [15:16:44] just short of paging at similar times in teh a.m. [15:17:20] PROBLEM - Puppet failure on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:18:08] chasemp: bblack: I had dos failure on "contintcloud" labs tenant if that matter [15:18:16] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [15:18:46] PROBLEM - Puppet failure on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:22:54] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [15:23:19] 6Labs: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2007920 (10Andrew) Happened again, also at 15:00 UTC. [15:23:37] 6Labs: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2007922 (10chasemp) Again today around Mon Feb 8 15:15:52 UTC 2016. I restarted pdns and pdns-recursor and it seems to have recovered it. Still unsure on the cause but the timing is pretty interesting. [15:29:38] 6Labs, 10Tool-Labs, 10Wikidata, 10Wikidata-Periodic-Table: /ptable project is broken - https://phabricator.wikimedia.org/T126223#2007934 (10ArthurPSmith) 3NEW [15:37:05] 6Labs: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2007971 (10jcrespo) The outage data happens at the same time than an increase in writes in a small subset of records on the m5-master (db1009) database. I cannot say if it is a cause or a consequence (e.g. it could be requests... [15:37:05] 6Labs: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2007973 (10scfc) Was the DNS server this time unresponsive or was it the "usual" temporary failure that self-righted? In the latter case, it could also be some form of network saturation, i. e. the error could be somewhere else. [15:45:12] 10MediaWiki-extensions-OpenStackManager, 10Notifications, 3Collaboration-Team-Current: Update OpenStackManager notifications to new language and format - https://phabricator.wikimedia.org/T125691#2008003 (10SBisson) a:3SBisson [15:49:51] 6Labs: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2008012 (10chasemp) @scfc, twice it has caught me by surprise when I'm not yet really awake but both times recovery corresponded w/ service restart which makes me think it's not network saturation. The service is doing somethi... [15:51:55] RECOVERY - Puppet failure on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:15] RECOVERY - Puppet failure on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:50] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:50] RECOVERY - Puppet failure on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:59:32] the server at discourse.search.eqiad.wmflabs is in the SHUTDOWN state, could anyone turn that back on for me? [15:59:47] and maybe help me figure out how it got there? not sure where to start looking [16:00:12] err, shutoff state (same idea) [16:12:42] (03CR) 10Awight: "@Legoktm" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/268794 (owner: 10Awight) [16:18:57] 6Labs, 5Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2008148 (10Andrew) - clients are not failing over to the backup dns server (holmium) at all. - There's a spike in pdns_concurrent queries at the time of the outage - The cache (as per metric pdns_cache - en... [16:21:43] 6Labs, 10Labs-Infrastructure, 6Phabricator: can't log in to phab-01.eqiad.wmflabs - https://phabricator.wikimedia.org/T125666#2008165 (10Negative24) I logged into phab-02 yesterday (actually I still am in tmux). `/tmp` is a 1 MiB partition that was full when I logged in with temp files from a mkinitramfs gen... [16:22:24] 6Labs, 5Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2008175 (10Andrew) "concurrent-queries shows the number of MThreads currently running" so that may be symptom rather than cause [16:24:35] 6Labs, 10Labs-Infrastructure, 6Phabricator: can't log in to phab-01.eqiad.wmflabs - https://phabricator.wikimedia.org/T125666#2008179 (10Dzahn) I got a this mail: Puppet is failing to run on the "**phab-03**" instance in the Wikimedia Labs project "phabricator" [16:27:43] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2008189 (10bd808) >>! In T122993#2007217, @EBernhardson wrote: > The regular expression that reads these doesn't like negative numbers. Just use a low %, like 10% or something... [16:30:08] 6Labs, 10Labs-Infrastructure, 6Phabricator: can't log in to phab-01.eqiad.wmflabs - https://phabricator.wikimedia.org/T125666#2008198 (10Dzahn) @yuvipanda how were you able to debug and what did you see? i also cant login as root on that instance. [16:36:55] 6Labs, 10Labs-Infrastructure, 6Phabricator: can't log in to phab-01.eqiad.wmflabs - https://phabricator.wikimedia.org/T125666#2008217 (10Negative24) Over two days I've gotten four email. Two about phab-03 and two about harbormaster1. [16:49:22] 6Labs, 10Labs-Infrastructure, 6Phabricator: can't log in to phab-01.eqiad.wmflabs - https://phabricator.wikimedia.org/T125666#2008248 (10Negative24) I'm trying to make sense of the Grafana disk space available graphs. I don't know specifically how the root's partition is fluctuating but it would be helpful t... [16:58:10] 6Labs, 5Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2008273 (10chasemp) p:5Triage>3High [17:09:33] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2008316 (10dcausse) This is actually a de-boost but you should maybe configure it to 1% or maybe 0% (might not be ideal: this will completely inhibit ranking if the purpose is... [17:16:39] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2008326 (10EBernhardson) yes those namespace filters are very odd, i've also just double checked and the namespace filters are only applied to web search, not api[1] which is... [17:48:51] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/NicoV was created, changed by NicoV link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/NicoV edit summary: Created page with "{{Tools Access Request |Justification=Use my tool WPCleaner[1] from Labs server: * to analyze dump files regularly without the need to download them on my computer first * e..." [17:49:54] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/NicoV was modified, changed by NicoV link https://wikitech.wikimedia.org/w/index.php?diff=291790 edit summary: copyedit [17:58:13] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2008534 (10bd808) >>! In T122993#2008316, @dcausse wrote: > If the goal is to remove nova resource from default search results could we simply remove this namespace from the d... [18:11:51] trying to apt-get update on phab-scap.eqiad.wmflabs: [18:11:54] Cannot initiate the connection to webproxy.eqiad.wmnet:8080 (2620:0:861:1:208:80:154:10). - connect (101: Network is unreachable) [IP: 2620:0:861:1:208:80:154:10 8080] [18:11:56] Fetched 229 kB in 2min 0s (1,905 B/s) [18:12:46] twentyafterfour, -> PM [18:13:34] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2008604 (10dcausse) Also, it looks like that deleted instances do not have the templates you've set in the boost-templates system message. These pages don't have any templates... [18:15:33] andrewbogott: who is fastcci-master.fastcci.eqiad.wmflabs? [18:15:57] I’ll look, just a minute [18:16:40] ok, try $ssh krenair@wikitech-static.wikimedia.org [18:16:45] passwd: omgchangeme [18:17:49] oops, wrong window, sorry :) [18:18:00] holy crap andrewbogott don't post that here [18:18:03] * andrewbogott changes that password in a hurry [18:18:06] I already changed it [18:18:15] ok, good :) [18:18:35] #1 IRC UI problem: people typing in the wrong channel [18:19:16] the question is, why is there a password-protected server anywhere [18:19:44] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2008643 (10EBernhardson) The namespace filters turned out to be a bug, any wiki with more than one content namespace was having the main namespace de-boosted by 95%. A default... [18:25:06] 6Labs, 10Tool-Labs, 10Wikidata, 10Wikidata-Periodic-Table: /ptable project is broken - https://phabricator.wikimedia.org/T126223#2008686 (10Johsthao) [18:25:32] 6Labs, 10Tool-Labs: tools-exec-12* puppet broken: php5* packages have been upgraded again? - https://phabricator.wikimedia.org/T126205#2008699 (10Johsthao) [18:26:00] 6Labs, 10Tool-Labs: tools-web-static-*: Could not find dependent Package[gridengine-common] - https://phabricator.wikimedia.org/T126171#2008712 (10Johsthao) [18:26:05] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2008716 (10Johsthao) [18:26:08] 6Labs, 10Tool-Labs: tools-docker-registry-01 has incorrect puppetmaster key - https://phabricator.wikimedia.org/T126167#2008714 (10Johsthao) [18:26:10] 6Labs, 10Tool-Labs: Toollabs::Cronrunner backup fails for invalid utf-8 content - https://phabricator.wikimedia.org/T126166#2008715 (10Johsthao) [18:31:35] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2008770 (10JEumerus) [18:31:37] 6Labs, 10Tool-Labs: Toollabs::Cronrunner backup fails for invalid utf-8 content - https://phabricator.wikimedia.org/T126166#2008768 (10JEumerus) 5duplicate>3Resolved [18:31:43] 6Labs, 10Tool-Labs: tools-docker-registry-01 has incorrect puppetmaster key - https://phabricator.wikimedia.org/T126167#2008771 (10JEumerus) 5duplicate>3Resolved [18:31:45] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2006444 (10JEumerus) [18:32:02] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2008776 (10JEumerus) 5duplicate>3Open [18:32:54] 6Labs, 10Tool-Labs, 10Wikidata, 10Wikidata-Periodic-Table: /ptable project is broken - https://phabricator.wikimedia.org/T126223#2008799 (10matmarex) 5duplicate>3Open [18:33:29] 6Labs, 10Tool-Labs: tools-exec-12* puppet broken: php5* packages have been upgraded again? - https://phabricator.wikimedia.org/T126205#2008819 (10matmarex) 5duplicate>3Open [18:33:59] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2008835 (10matmarex) [18:34:01] 6Labs, 10Tool-Labs: tools-web-static-*: Could not find dependent Package[gridengine-common] - https://phabricator.wikimedia.org/T126171#2008834 (10matmarex) 5duplicate>3Open [18:34:03] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2006444 (10matmarex) [18:34:13] 6Labs, 10Tool-Labs: tools-docker-registry-01 has incorrect puppetmaster key - https://phabricator.wikimedia.org/T126167#2008836 (10matmarex) 5Resolved>3Open [18:34:15] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2006444 (10matmarex) [18:34:17] 6Labs, 10Tool-Labs: Toollabs::Cronrunner backup fails for invalid utf-8 content - https://phabricator.wikimedia.org/T126166#2008837 (10matmarex) 5Resolved>3Open [18:38:08] 6Labs, 10Labs-Infrastructure: Get HA db support for labs services - https://phabricator.wikimedia.org/T126251#2008867 (10Andrew) 3NEW a:3jcrespo [18:44:14] if any labs admins have a moment, can you look into how https://wikitech.wikimedia.org/wiki/Nova_Resource:Discourse.search.eqiad.wmflabs ended up in the stopped state, or perhaps just restart it? [18:52:42] 6Labs, 10wikitech.wikimedia.org: Have a process for regularly updating wikitech-static - https://phabricator.wikimedia.org/T125709#2009003 (10Krenair) [19:02:38] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ad Huikeshoven was created, changed by Ad Huikeshoven link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Ad_Huikeshoven edit summary: Created page with "{{Tools Access Request |Justification=There is a Discourse installation at https://discourse.wmflabs.org/ I happen to be admin on that installation. I'm trying to figure thing..." [19:07:18] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ad Huikeshoven was modified, changed by Ad Huikeshoven link https://wikitech.wikimedia.org/w/index.php?diff=291972 edit summary: [19:13:22] Hi My name is Ad Huikeshoven https://discourse.wmflabs.org/ is still down, produces a 502 Bad Gateway error ebernhardson [19:14:02] dedalus_: discourse.wmflabs.org is not hosted on tool labs (although I'd be happy to give you access to TL) [19:14:29] ebernhardson: mentioned the server hosting it is in some weird state [19:16:12] dedalus_: havn't had any luck getting ahold of a labs admin yet, but i filled a ticket at https://phabricator.wikimedia.org/T126191 for them to look into it and restart the instance, also brought it up here a couple times but no luck yet [19:19:38] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 2 others: Move project membership/assignment from ldap to keystone mysql - https://phabricator.wikimedia.org/T115029#2009079 (10Andrew) [] disable wikitech logins unset( $wgSpecialPages['UserLogin'] ); in wikitech.php [] log out a... [19:22:22] 6Labs, 10Tool-Labs: Toollabs::Cronrunner backup fails for invalid utf-8 content - https://phabricator.wikimedia.org/T126166#2009102 (10scfc) 5Open>3Resolved [19:22:24] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2009103 (10scfc) [19:23:07] 6Labs, 10Tool-Labs: tools-docker-registry-01 has incorrect puppetmaster key - https://phabricator.wikimedia.org/T126167#2009109 (10scfc) 5Open>3Resolved [19:23:08] 6Labs, 10Tool-Labs: puppet failure on a large number of instances - https://phabricator.wikimedia.org/T126165#2006444 (10scfc) [19:23:25] valhallasw`cloud: ebernhardson thanks for the replies [19:24:50] Hi multichill can you do something with https://discourse.wmflabs.org/ ? [19:25:14] dedalus_: nee, een van de ops labs admins moet er naar kijken [19:25:31] valhallasw`cloud: ok [19:25:58] the server is in an odd error state that requires someone to login to the backend server [19:26:39] * dedalus_ as an admin I upgraded the installation last night. It worked after the upgrade, I checked ... [19:26:59] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 2 others: Move project membership/assignment from ldap to keystone mysql - https://phabricator.wikimedia.org/T115029#2009137 (10Andrew) to revert, roll back all patches, reset all caches, and truncate keystone tables: assignment, proj... [19:36:41] dedalus_: in theory doing work inside the instance shouldn't cause the cluster to shut the instance off, but not sure yet why the cluster stopped the instance (i don't have access to any logs like that) [19:37:50] ebernhardson: just to be sure, the 'reboot' button on wikitech doesn't solve this, right? [19:38:42] valhallasw`cloud: right, it did nothing :( [19:43:03] andrewbogott: ^ can you take a look at https://wikitech.wikimedia.org/wiki/Nova_Resource:Discourse.search.eqiad.wmflabs ? [19:43:23] it's shut down for some reason, and wikitech doesn't allow users to boot shut down instances :/ [19:43:55] valhallasw`cloud: yes, remind me in a few minutes [19:44:08] * valhallasw`cloud nods [19:58:00] valhallasw`cloud, Hello :) I do not know if you remember, I am tatoo, some time came for help on wikibugs, I have some doubts, can you help me? is a configuration... [19:58:17] kwargs: hey! yes, of course. [19:58:21] 6Labs, 10wikitech.wikimedia.org: Have a process for regularly updating wikitech-static - https://phabricator.wikimedia.org/T125709#2009341 (10Krenair) a:5Andrew>3Krenair [20:00:10] ebernhardson: valhallasw`cloud hey sorry we are running a bit crazy this week, short handed etc [20:01:55] valhallasw`cloud, in settings: mediawiki_user ... the user must be mediawiki.org or our wiki? [20:02:41] kwargs: eeeh. I'm not entirely sure what it's used for, actually :-) let me check [20:03:09] kwargs: ah, it's used for taxonomy.py which dumps all projects to a wiki page [20:03:48] kwargs: so you can probably just set that to None; if you want to use the taxonomy, you'd have to adapt taxonomy.py [20:03:59] kwargs: https://www.mediawiki.org/wiki/Phabricator/Projects [20:08:31] valhallasw`cloud, Oh, really thank you very much :) [20:09:11] kwargs: sorry, it's all pretty wikimedia-centric, but we'd be happy to merge improvements to make it work better for third parties :-) [20:17:25] andrewbogott: reminder after a few minutes about discourse.search.eqiad.wmflabs :) [20:18:29] ebernhardson: I started it [20:18:35] * YuviPanda should go afk for vacation soon [20:18:43] YuviPanda: thanks :) any ideas why it shut off [20:18:46] nope :) [20:19:08] * YuviPanda leaves IRC now [20:19:10] good bye :) [20:19:14] bye, thanks! [20:34:36] so weird thing, i no longer have root sudo access on discourse.search.eqiad.wmflabs [20:35:19] same on other instances in search.eqiad.wmflabs [20:36:31] on the instances that run mwvagrant though i can sudo to the vagrant user, then sudo to root [20:37:00] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2009518 (10MusikAnimal) @TheDJ Fixed, though I'm not able to actually get any data from the pageviews API. Are we sure wikidata is supported? @Sjoerddebruin I a... [20:37:12] How could I claim my @toolserver.org address (need to register it phabricator) [20:37:42] Dispenser: please file a bug in phabricator [20:37:44] I may have blanked it for privacy reasons when the Toolserver shut down [20:38:25] ebernhardson: let me take a look... [20:39:44] https://www.irccloud.com/pastebin/r8qo5kN1/ [20:40:03] ebernhardson: is it OK if I run puppet? [20:40:06] valhallasw`cloud: yup [20:40:21] valhallasw`cloud: but same issue on cirrus-browser-bot.search.eqiad.wmflabs, which has had a puppet run in the last 20 minutes [20:40:46] yeah, puppet doesn't do anything [20:42:36] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2009523 (10Sjoerddebruin) >>! In T120497#2009518, @MusikAnimal wrote: > @TheDJ Fixed, though I'm not able to actually get any data from the pageviews API. Are we... [20:45:09] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2009528 (10MusikAnimal) @Sjoerddebruin yes, there is an issue with ad blockers. I have no idea why, but there definitely aren't any ads :) I put a notice about t... [20:45:38] ebernhardson: hrm. I'm not really sure what to check :/ it's clearly pam_ldap that's doesn't want to work, but I don't know the details of how that works [20:47:31] ldaplist works as expected, so it can talk to ldap [20:47:38] which is also clear from the fact that you can login [20:49:16] ebernhardson: all users in project-search have sudo already, right? [20:49:37] hm, no, that's a different list [20:50:54] if i had to guess, there is supposed to be a `$project-search ALL=(ALL) NOPASSWD: ALL` in /etc/sudoers.d/somefile but it's missing [20:51:02] s/$/% [20:51:26] except that not everyone in project-search is also an admin according to wikitech [20:51:37] in any case, I have now added an explicity sudo entry for you [20:51:52] valhallasw`cloud: thanks, it is quite an odd thing ... [20:52:18] ebernhardson: can you test if it works? [20:53:04] Hi - anyone there who is familar with https://analytics.wmflabs.org/demo/pageview-api/ ? [20:53:12] valhallasw`cloud: works now with the explicit right [20:53:21] ebernhardson: ok! I'll make a phab ticket... [20:53:58] YuviPanda: thanks! [20:54:30] 6Labs: sudo does not work for admin users in 'search' project - https://phabricator.wikimedia.org/T126265#2009576 (10valhallasw) 3NEW [20:55:00] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2009583 (10MusikAnimal) @Sjoerddebruin so you did see the notice, then? which ad blocker extension are you using? I've got to get to the bottom of this. [20:55:12] YuviPanda: can you help with https://analytics.wmflabs.org/demo/pageview-api/ ? There are umlaut problems ... [20:55:58] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2009589 (10Sjoerddebruin) >>! In T120497#2009583, @MusikAnimal wrote: > @Sjoerddebruin so you did see the notice, then? which ad blocker extension are you using?... [20:57:21] doctaxon, Partynia, discuss it in that bug? [20:57:55] does a bug exists? [20:58:03] https://phabricator.wikimedia.org/T120497#2009589 [20:58:09] is where people seem to be discussing that tool? [20:58:14] search doesn`worl, because ä, ö, ü are not recognized [20:58:20] work [20:58:31] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2009600 (10MusikAnimal) @Sjoerddebruin could you link me to the extension? Not seeing it at https://safari-extensions.apple.com/?q=ublock [20:58:44] Partynia: search where? [20:58:45] ü = %C3%BC [20:58:49] yes [20:58:54] I need to fix [20:59:01] Partynia: yes, that's correct. [20:59:07] https://analytics.wmflabs.org/demo/pageview-api/ [20:59:09] ebernhardson: in the pageview stats tool [20:59:16] or wait, is it the API that's broken? [20:59:18] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2009601 (10Sjoerddebruin) >>! In T120497#2009600, @MusikAnimal wrote: > @Sjoerddebruin could you link me to the extension? Not seeing it at https://safari-extens... [20:59:19] oh ok, not the search i'm responsible for :) [20:59:21] MusikAnimal: it's the api [20:59:27] ok good ha [20:59:30] MusikAnimal: https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/%C3%9Cber/daily/2016011900/2016020700 [20:59:38] err, not good, but not my fault :) [20:59:53] glad I saw this. Just had someone ask about the same thing [21:00:01] oh, actually that does call my search :S [21:00:38] although, pasting ü into the query seems to work [21:00:49] yeah [21:00:53] I noticed that as well [21:00:54] MusikAnimal: if you could fix it this will be fine [21:01:26] I'll have to blacklist which kinds of characters should be encoded [21:01:33] not fun [21:01:52] ya, not fun [21:02:06] MusikAnimal: eh, as far as I can see they are encoded correctly? [21:02:06] I'll figure it out, kind of important for the non-English projects [21:02:33] valhallasw`cloud: well, it's the API that doesn't accept the encoded URI right? [21:03:18] MusikAnimal: I'm not sure what you mean. The API does not give a correct response when you pass a correctly-encoded 'Über', no. [21:03:20] MusikAnimal: the other thing, it looks like this us using full text search for autocomplete? If you used https://en.wikipedia.org/w/api.php?callback=articleSuggestionCallback&action=query&list=prefixsearch&format=json&pssearch=don latency should be cut in half (would be more, but by then latency is dominated by the internet and mediawiki api wrapper) [21:03:24] I use the History API to push states to the browser, so you'll have deeplinking. That needs to be encoded, but evidently the foreign characters should not be [21:03:54] also Über does not actually exist (it's deleted) [21:03:56] ebernhardson: I think that one doesn't let you search for redirects [21:04:01] which is why I changed it [21:04:12] oh, wait, enwiki not dewiki [21:04:17] it seems to go pretty fast though [21:04:37] YuviPanda: awake ? [21:05:04] MusikAnimal: prefix search returns redirects [21:05:33] MusikAnimal: https://analytics.wmflabs.org/demo/pageview-api/ on dewiki w/ übersetzer seems to work, also with autocomplete? [21:05:40] for exampel https://en.wikipedia.org/w/api.php?callback=articleSuggestionCallback&action=query&list=prefixsearch&format=json&pssearch=albert+cuy return albert cuyp, which is a redirect to aelbert cuyp [21:05:45] ebernhardson: so it does! I will update soon [21:05:51] not sure what I was using before [21:06:36] MusikAnimal: you will then also get the new completion suggester (does fuzzy queries) as we roll it out to prod :) You can force the new algorithm with `cirrusUseCompletionSuggester=yes` in the query (after this weeks deployment train rolls forward) [21:19:35] ebernhardson: I will play around with that, thank you! [21:19:49] I've been overwhelmed with requests for this project [21:19:55] happens with popular things :) [21:20:04] I just took the demo and added deep linking, then discovered how much people wanted it [21:21:12] get about 4K hits a day [21:22:31] 6Labs: Instance discourse.search.eqiad.wmflabs in SHUTDOWN state - https://phabricator.wikimedia.org/T126191#2009654 (10AdHuikeshoven) 5Open>3Resolved [21:23:47] Thank you for your help ! [21:23:58] 10Labs-Other-Projects: Succesful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#2009665 (10AdHuikeshoven) >>! In T124690#2007048, @Samwilson wrote: > https://discourse.wmflabs.org/ is currently down: 502 Bad Gateway T... [21:27:58] (03CR) 10Legoktm: [C: 032] Send Education-* to wikimedia-ed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/268794 (owner: 10Awight) [21:36:06] 10Labs-Other-Projects: Succesful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#2009784 (10AdHuikeshoven) [22:07:40] 6Labs: sudo does not work for admin users in 'search' project - https://phabricator.wikimedia.org/T126265#2009922 (10scfc) Has the Search project a sudoers policy for roots at https://wikitech.wikimedia.org/wiki/Special:NovaSudoer (named "roots" for example in the Tools project)? Its "Allow running as" should b... [22:09:47] (03Merged) 10jenkins-bot: Send Education-* to wikimedia-ed [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/268794 (owner: 10Awight) [22:10:52] !log tools.wikibugs Updated channels.yaml to: 2dd0d574c0e2bfcd5285493664f884f2ddc54b99 Send Education-* to wikimedia-ed [22:10:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [22:39:01] andrewbogott: hey Andrew, awight is trying to add me to the globaleducation project admins, but he apparently doesn't see "labs projectadmins" on wikitech. is there another layer of permissions or something? [22:39:12] afaik, he's already a projectadmin [22:39:29] I will look [22:39:38] rad, thanks! [22:40:01] did awight try logging out and in already? [22:40:34] hmm, not sure. i told him to verify that he was logged in but i'll tell him to re-auth [22:40:46] marxarelli, what is your username on wikitech? [22:40:59] Hi! I'm logged in as "Awight" if this is about me :) [22:41:03] andrewbogott: dduvall [22:41:19] marxarelli: ok, I added you [22:41:26] andrewbogott: \o/ thanks! [23:14:21] PROBLEM - SSH on tools-exec-1217 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:33] hi [23:20:05] anyone here? [23:20:05] Hi Peter__, I am here, if you need anything, please ask, otherwise no one is going to help you... Thank you [23:34:19] 6Labs, 10wikitech.wikimedia.org: Exclude nova resource pages from *default* wikitech search - https://phabricator.wikimedia.org/T122993#2010152 (10Tgr) >>! In T122993#2008534, @bd808 wrote: > We do want to squash the instance pages however which the template based de-boost should mostly do. I've set them to 1%... [23:37:43] 6Labs, 10wikitech.wikimedia.org: Add links to Labs help/FAQ on Nova Resource project and instance pages - https://phabricator.wikimedia.org/T126289#2010164 (10bd808) 3NEW [23:38:14] 6Labs, 10wikitech.wikimedia.org: Add links to Labs help/FAQ on Nova Resource project and instance pages - https://phabricator.wikimedia.org/T126289#2010174 (10bd808) [23:38:16] 6Labs, 6Developer-Relations, 10wikitech.wikimedia.org, 7Epic: [EPIC] Make wikitech more friendly for the multiple audiences it supports - https://phabricator.wikimedia.org/T123425#2010173 (10bd808) [23:46:21] noone here [23:46:23] ? [23:51:32] Peter__: Hello! What friendly wm-bot was suggesting is that you should go ahead and ask your question. Nobody knows whether they can help you, without hearing what you need... [23:52:50] oops ok [23:53:18] but it is a little to specific [23:53:28] Perfect ;) [23:53:43] are you working as a developer for wiki? [23:53:57] i would like to give you an idea :) [23:55:32] Peter__: hehe. I am in fact, but it's probably not like you imagine--if you have a feature idea, try suggesting it on http://phabricator.wikimedia.org/ [23:57:52] anyway may i ask you here if it is likely that wiki is interested in building this up? because if not, theres no need to descrie it in a long mail :) [23:58:34] You can certainly ask anything you'd like!