[00:14:12] 06Labs, 10Labs-Infrastructure, 10Icinga, 06Operations: remove/fix "Check for gridmaster host resolution" Icinga check for "labtest" - https://phabricator.wikimedia.org/T152024#3177166 (10Dzahn) >>! In T152024#2835945, @Krenair wrote: > "gridmaster host resolution" is a tools project specific thing, why is... [00:17:17] 06Labs, 10Labs-Infrastructure, 10Icinga, 06Operations: remove/fix "Check for gridmaster host resolution" Icinga check for "labtest" - https://phabricator.wikimedia.org/T152024#3177189 (10Dzahn) Ah, there is a section in hieradata/regex.yaml for labtest-specific settings. That is also what disables paging.... [06:48:22] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:28:22] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:51:05] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:26:06] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [09:12:43] Hi. I am trying to add a user to a group in gerrit, but I cannot find the user in LDAP although there's a wikitech User page for the user. Any ldap guru that could help me troubleshoot? (Will pm username) [09:50:33] PROBLEM - Puppet run on tools-exec-1434 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:51:22] (03CR) 10Dereckson: "Perhaps the readme should reflect the *current* state of code base instead of what's coming." [labs/tools/Wikimedia-Emoji-Bot] - 10https://gerrit.wikimedia.org/r/348010 (owner: 10D3r1ck01) [10:25:34] RECOVERY - Puppet run on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Valerio Bozzolan was created, changed by Valerio Bozzolan link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Valerio_Bozzolan edit summary: Created page with "{{Tools Access Request |Justification=To add an it.wiki Leaflet user map (in order to deprecate Google Mashup). |Completed=false |User Name=Valerio Bozzolan }}" [13:17:05] qchris_: what is the the user and wikitech page you are looking at? [13:17:17] (pm is fine) [13:47:06] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:16:16] chasemp: Thanks. Sent the user page in pm. [14:27:07] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [15:09:01] test [15:26:40] chasemp: I received an email from fantasticfears (T162134, I kind of know him IRL) about him being unable to ssh into his instance. After a bit of debugging he doesn't seem to be in the LDAP group 'project-discourse-wam'. Could that be a reason? [15:26:40] T162134: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134 [15:28:03] zhuyifei1999_: hm [15:28:05] PROBLEM - Puppet run on tools-exec-1432 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:28:50] zhuyifei1999_: andrewbogott so fantasticfears shows up for me as a projectadmin via wikitech [15:28:55] but [15:29:13] getent group project-discourse-wam [15:29:30] root@tools-exec-1432:/var/log# id fantasticfears [15:29:30] uid=3229(fantasticfears) gid=500(wikidev) groups=50062(project-bastion),50380(project-tools),500(wikidev) [15:29:49] huh [15:29:50] andrewbogott: do we have some keystone vs ldap inconsisencies going on? [15:30:05] maybe. I'll poke at it [15:30:08] andrewbogott: this project was created yesterday so there may be a bug in some new project creation handling? [15:30:14] zhuyifei1999_: tahnks for letting us know^ [15:30:30] https://tools.wmflabs.org/contact/search/project-discourse-wam nothing whereas https://tools.wmflabs.org/contact/search/project-video works fine [15:30:33] np [15:31:31] chasemp: you created the new project with wikitech? [15:33:04] RECOVERY - Puppet run on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [15:33:32] andrewbogott: yes [15:33:34] https://phabricator.wikimedia.org/F7529006 [15:33:37] same as usual [15:33:57] (my wifi is flapping here fyi) [15:36:03] something is definitely broken with new project creation; zhuyifei1999_ I'll let you know when I have it sorted out. [15:36:08] k [15:50:50] RECOVERY - Host tools-exec-1433 is UP: PING OK - Packet loss = 0%, RTA = 2.01 ms [15:52:03] ^ did we know that was ...down? [15:52:42] 06Labs, 10Labs-Infrastructure, 10Icinga, 06Operations, 13Patch-For-Review: remove/fix "Check for gridmaster host resolution" Icinga check for "labtest" - https://phabricator.wikimedia.org/T152024#3178867 (10Dzahn) 05Open>03Resolved Thanks for merging @Andrew and it's gone from https://icinga.wikime... [15:59:01] PROBLEM - Host tools-exec-1433 is DOWN: CRITICAL - Host Unreachable (10.68.22.87) [16:26:12] (03CR) 10Thcipriani: [C: 031] Scap: Remove git_server from scap.cfg [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/347926 (https://phabricator.wikimedia.org/T162814) (owner: 10Thcipriani) [17:03:11] paladox: are you working in the phabricator project? It's sending a million emails that are bouncing and spamming us. [17:03:21] e.g. [17:03:24] oh [17:03:33] https://www.irccloud.com/pastebin/Imf3LfOt/ [17:03:46] oh [17:03:49] how do i disable that [17:03:59] no idea [17:05:26] oh [17:06:45] I see why it's doing that as i use the production puppet module. It enables email notification. [17:06:54] I've stoped exim4 though it may re enable it's if i reboot or puppet may do it. [17:07:14] i see this [17:07:14] ALERT: exim paniclog /var/log/exim4/paniclog has non-zero size, mail system possibly broken [17:07:20] andrewbogott ^^ [17:07:37] 2017-03-12 21:45:05 Exim configuration error for smart_route router: [17:07:37] route_list or route_data (but not both) must be specified [17:07:52] paladox: Maybe you can configure it to send email to an address that exists? [17:08:02] Oh. [17:09:00] i wonder if that is done in [17:09:00] hostlist wikimedia_nets = <; 91.198.174.0/24 ; 208.80.152.0/22 ; 2620:0:860::/46 ; 198.35.26.0/23 ; 185.15.56.0/22 ; 2a02:ec80::/32 ; 10.0.0.0/8 [17:09:00] hostlist relay_from_hosts = <; @[] ; 127.0.0.1 ; ::1 ; [17:11:25] i guess it's comming from [17:11:25] never_users = root : daemon : bin [17:17:19] andrewbogott is there a way to stop phabricator from filling your mail box [17:17:32] Sorry that it was doing that. Didnt even realise that. [17:17:52] paladox: I don't know anything about phabricator. Surely you can turn it off though. [17:18:20] I doint know how to as it is the phabricator class that does it. though i've stopped it. [17:18:26] how do i mask it [17:18:30] exim4 [17:18:48] probably the email address (or the email switch) is in a config file, if you're lucky it's configured from hiera. [17:19:16] Oh, i will ask twentyafterfour who will know :) [17:54:56] paladox: probably best to kill exim and disable puppet until you have a proper fix [17:55:09] oh [17:56:07] andrewbogott im wondering did it stop spamming you as soon as i stopped the service? [17:56:13] If not it may be something else. [17:56:23] I don't know, I'm not getting the messages, just hearing other people complain about them [17:57:07] oh [17:57:08] ok [17:58:10] anyways i masked it. So the service wont stop. I verfied puppet wont overide this by running it. So hopefully the emails wont get sent now [18:09:16] Hi there. One of our tools is facing database corruption. I wonder if you have seen this error before: [18:09:21] https://www.irccloud.com/pastebin/wji5Vf7k/ [18:11:08] Niharika: I have not seem that at all, seems worth a task [18:11:11] s/seem/seen [18:11:25] chasemp: Okay, I'll file one. [18:11:34] andrewbogott apparently it was cron [18:12:20] anyways the rules were disabled. So the spam should not happen now. I didnt know cron sent emails out when running commands. [18:13:16] re - db corruption, possibly related: https://phabricator.wikimedia.org/T130128 [18:13:19] (03CR) 10BryanDavis: [C: 032] Scap: Remove git_server from scap.cfg [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/347926 (https://phabricator.wikimedia.org/T162814) (owner: 10Thcipriani) [18:13:25] (03Merged) 10jenkins-bot: Scap: Remove git_server from scap.cfg [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/347926 (https://phabricator.wikimedia.org/T162814) (owner: 10Thcipriani) [18:15:25] 06Labs, 10Tool-Labs: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179794 (10Niharika) [18:16:59] 06Labs, 10Tool-Labs, 06Community-Tech: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179819 (10Niharika) [18:17:53] paladox: generally cron will email stdout of any command that exits with a non-success status to the crontab owner. prod puppet roles wouldn't suppress that generally because if things break in prod we want to know about it. in labs it can cause all kinds of problems if you aren't careful [18:18:28] oh ok [18:18:40] 06Labs, 10Tool-Labs, 06Community-Tech: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179828 (10MusikAnimal) p:05Triage>03High [18:24:07] chasemp, Zhaofeng_Li, I'm going to delete and recreate that project, hopefully it will work better this time. [18:26:10] 06Labs, 10Tool-Labs, 06Community-Tech: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179794 (10MusikAnimal) (Possibly unbreak now?) Apparently this only happens when you `ORDER BY diff_timestamp`. One quick and dirty workaround is to order by `id`, which should in theory should be... [18:27:19] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179864 (10bd808) [18:30:42] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179885 (10MusikAnimal) I changed the query in CopyPatrol and it is at least functional now: https://tools.wmflabs.org/copypatrol/en [18:30:54] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3179888 (10chasemp) [18:33:30] Zhaofeng_Li, should be fixed now [18:33:59] 06Labs: Request creation of Discourse for Wiki Asian Month labs project - https://phabricator.wikimedia.org/T162134#3179896 (10Andrew) Apologies, there was a bug in the project creation code. I've deleted and recreated the project, it should be working properly now. Please let me know if not! [18:46:43] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3180146 (10MusikAnimal) [18:46:45] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: labvirt1001 and 1002 cannot launch new VMs - https://phabricator.wikimedia.org/T159721#3180149 (10Andrew) [18:46:47] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad, 13Patch-For-Review: Labvirt1001 has insanely slow IO - https://phabricator.wikimedia.org/T159835#3180147 (10Andrew) 05Open>03Resolved Repooled, seems fine. [18:46:51] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3179794 (10MusikAnimal) [18:47:13] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3180172 (10MusikAnimal) [19:00:39] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3180313 (10MusikAnimal) Hmm seems to be working now? [19:02:13] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3180323 (10MusikAnimal) p:05High>03Normal [19:12:35] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3180464 (10chasemp) I noticed we saw puppet failures from tools-exec-1432 this morning so I decided to take a look. I untangled the layers in shinken to recreate the command that demonstrate... [19:20:12] 06Labs, 10Horizon: keystonehooks: Figure out about member role removal - https://phabricator.wikimedia.org/T162615#3180521 (10Andrew) The upstream is in ocata, we need local workarounds until then. [19:39:43] 06Labs: IO issues for Tools instances flapping with iowait and puppet failure - https://phabricator.wikimedia.org/T161898#3146683 (10bd808) I filed https://github.com/wsexport/tool/issues/127 with the wsexport tool. [19:40:14] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3180659 (10Ragesoss) p:05Normal>03High Not any more. [20:11:44] 06Labs, 06Operations, 10hardware-requests: eqiad: (1) hardware access request for labnodepool1002 - https://phabricator.wikimedia.org/T161753#3141989 (10RobH) a:03RobH Ok, this one is slightly confusing when taken in with all the other labs requests, so I had to put all of them in to a single spreadsheet w... [20:12:15] 06Labs, 06Operations, 10hardware-requests: Codfw: (2) hardware access request for labtest [region 2] - https://phabricator.wikimedia.org/T161766#3142263 (10RobH) Ok, this one is slightly confusing when taken in with all the other labs requests, so I had to put all of them in to a single spreadsheet with the... [20:12:51] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestneutron refresh - https://phabricator.wikimedia.org/T154706#2921133 (10RobH) Ok, this one is slightly confusing when taken in with all the other labs requests, so I had to put all of them in to a single spreadsheet with... [20:13:30] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestnet2003 [region 2] - https://phabricator.wikimedia.org/T161764#3142232 (10RobH) Ok, this one is slightly confusing when taken in with all the other labs requests, so I had to put all of them in to a single spreadsheet wi... [20:14:09] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labcontrol1003/1004 - https://phabricator.wikimedia.org/T158207#3029754 (10RobH) Ok, this one is slightly confusing when taken in with all the other labs requests, so I had to put all of them in to a single spreadsheet with the... [20:14:36] 06Labs, 06Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestvirt2003 [region 2] - https://phabricator.wikimedia.org/T161765#3142249 (10RobH) Ok, this one is slightly confusing when taken in with all the other labs requests, so I had to put all of them in to a single spreadsheet w... [20:47:50] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3180979 (10Andrew) [20:50:32] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3180983 (10Andrew) [20:51:06] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3180968 (10Andrew) [20:59:01] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3181017 (10Andrew) [21:01:10] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3181031 (10Andrew) I cleaned up a few obviously-unused groups and deleted them from the lists above. I also emailed @petrb to ask about the groups... [21:05:02] 10Labs-project-other: Can I delete the Labs 'orgcharts' project? - https://phabricator.wikimedia.org/T162947#3181042 (10Andrew) [21:05:08] 10Labs-project-other: Can I delete the Labs 'orgcharts' project? - https://phabricator.wikimedia.org/T162947#3181055 (10Andrew) p:05Triage>03Low [21:11:31] halfak: I see that you (in the persona of snuggle) are in the rare company of people who used service groups outside of tools. Is that a thing you consider useful and/or necessary? [21:11:37] (this is re: https://phabricator.wikimedia.org/T162945 [21:11:38] ) [21:11:39] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3181114 (10bd808) [21:11:43] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic: Tools web interface for tool authors (Brainstorming ticket) - https://phabricator.wikimedia.org/T128158#3181113 (10bd808) [21:12:04] hey quick question to be added as a maintainer for something in labs but you only have requested access for tools could you still be added as a maintainer in labs without needing to request additonal access? [21:12:19] andrewbogott, I'm not sure I remember why I'm using a service group. [21:12:30] It might have happened while I was trying to figure out how to set permissions and stuff. [21:12:57] what *is* a service group outside of Tools? [21:13:12] Zppix: once you can log in to tools, you have all the necessary access to reach any other project that adds you. [21:13:28] halfak: it's just a posix group and user [21:13:39] andrewbogott: thanks i just verifying [21:14:04] andrewbogott, I'm going to go with "user error" as my reason for using a service group them :) [21:14:17] halfak: so, in your case you have two groups, 'snuggle.database' and 'snuggle.www-data' [21:14:21] each of which contain only you as a member [21:15:00] halfak: if you're sure, I'll just delete them and cross them off my list of 'mysterious uses of this feature' [21:15:12] +1 [21:15:30] thanks! [21:15:40] putmantime, hi! [21:15:45] I just saw that you connected. [21:15:51] Hello! [21:15:55] Want to talk about OAUTH? [21:16:32] halfak: deleted! [21:16:38] OMG WAIT! [21:16:41] j/k <3 [21:16:42] :D [21:17:13] Yes that would be great. I am trying to connect my consumer app with a specified callback url and for some reason it wont redirect back and I get a E002 error. [21:17:40] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3181181 (10Andrew) [21:19:18] putmantime, I'm not sure what that error is. [21:19:23] DO you have any more information? [21:19:32] Also, I'd like to see how you're configuring OAuth [21:19:37] I have an idea for a test too :) [21:19:38] https://www.mediawiki.org/wiki/Help:OAuth/Errors#E002 [21:20:03] Yes, after hitting allow. The browser attempts https://www.mediawiki.org/wiki/Special:OAuth/www.chlambase.org/organism/471472/gene/CTL0007?oauth_verifier=44c6a58c5a5f865837ee3f30ec1fb29b&oauth_token=ae0de10f3498eaed21741298acaa48bf [21:20:22] https://www.mediawiki.org/wiki/Help:OAuth/Errors#E002 [21:20:28] No URL of the OAuth app has been given. There is probably an issue in the code logic of the external application. [21:21:06] putmantime, ! [21:21:19] Put an http in front of your callback! [21:21:22] It's not a URL [21:21:23] :P [21:21:34] URLs have a protocol :) [21:21:45] * halfak is very happy this might be a simple fix. [21:21:52] That is what we were just contemplating. Thanks. I'm kinda new at this [21:21:54] also, I highly recommend putting a "/" at the end of your callback URL. [21:22:04] No problem putmantime, this stuff is confusing :) [21:22:41] Ok, so I will propose a new version with the protocol and a trailing /. Thank you! [21:23:34] 10Labs-project-other: Is the 'webplatform' labs project defunct? - https://phabricator.wikimedia.org/T162951#3181213 (10Andrew) [21:23:40] Happy to help :) Feel free to ping me any time in this channel if you need help again. [21:23:46] also messaging me on-wiki is fine too :D [21:24:04] Thank you! Cheers! [21:30:49] 10Labs-project-other: Can I delete the Labs 'orgcharts' project? - https://phabricator.wikimedia.org/T162947#3181230 (10Matanya) Ok from my side. I don't have time to work on it. [21:31:33] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs: The future of service groups and service users on Labs - https://phabricator.wikimedia.org/T162945#3181231 (10Andrew) orgcharts and webplatform seem to be defunct. So, outside of bots the remaining possible users are: cn=catgraph.catgraph,ou=ser... [21:31:35] 10Labs-project-other: Can I delete the Labs 'orgcharts' project? - https://phabricator.wikimedia.org/T162947#3181232 (10MarkTraceur) Yeah, that project is long gone. [21:51:02] 06Labs, 06Operations: rebuild tools-grid-master as a large instance - https://phabricator.wikimedia.org/T162955#3181356 (10chasemp) [21:57:26] 06Labs, 10Labs-Infrastructure: Monitor dhcp/dnsmasq on labnet - https://phabricator.wikimedia.org/T162956#3181386 (10Andrew) [22:00:27] 06Labs, 06Operations: rebuild tools-grid-master as a large instance - https://phabricator.wikimedia.org/T162955#3181425 (10chasemp) If this was successful I wonder if we could easily bake in http://wiki.gridengine.info/wiki/index.php/RQS_Common_Uses#Max_user_jobs_in_a_particular_queue [22:00:30] 10Labs-project-other: Can I delete the Labs 'orgcharts' project? - https://phabricator.wikimedia.org/T162947#3181426 (10Andrew) 05Open>03Resolved Thanks for the quick response, y'all! [22:18:53] 06Labs, 06Developer-Relations (Apr-Jun 2017): Recover Zulip/Mattermost instance on Labs - https://phabricator.wikimedia.org/T162960#3181495 (10srishakatux) [22:19:09] 06Labs, 06Developer-Relations (Apr-Jun 2017): Recover Zulip/Mattermost instance on Labs - https://phabricator.wikimedia.org/T162960#3181508 (10srishakatux) p:05Triage>03High [22:19:45] 06Labs, 06Developer-Relations (Apr-Jun 2017): Recover Zulip/Mattermost instance on Labs - https://phabricator.wikimedia.org/T162960#3181495 (10srishakatux) [22:20:18] !log tools.framabot webservice stop; webservice start [22:20:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.framabot/SAL [22:23:55] !log tools.framabot mv .lighttpd.conf .lighttpd.conf-broken [22:23:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.framabot/SAL [22:41:09] 06Labs, 10Tool-Labs: Limit webservice manifest restarts - https://phabricator.wikimedia.org/T107878#3181566 (10bd808) p:05Normal>03High framabot was in an infinite loop of restarts (28799 in the last 7 days) because of a bad `.lighttpd.conf` file. We need to do something at least like bigbrother's 10 and d... [22:59:07] !log wikilabels staged wikilabels-wmflabs-deploy:9f35ab6 [22:59:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [23:06:54] !log wikilabels staged wikilabels-wmflabs-deploy:e96b3da [23:06:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [23:16:50] !log wikilabels staged wikilabels-wmflabs-deploy:326732d [23:16:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [23:19:08] !log wikilabels deployed wikilabels-wmflabs-deploy:326732d [23:19:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [23:19:55] \o/ We have a localized (or localizable) message at the top of wikilabels! Now we're going to need to figure out how to read headers to identify ltr/rtl prefs. :) [23:22:29] OK. I'm out of here! Have a good one, folks :) [23:46:22] 06Labs, 10Tool-Labs, 06Community-Tech, 10DBA: Fix Plagiabot DB corruption - https://phabricator.wikimedia.org/T162932#3181900 (10jcrespo) 05Open>03Invalid Cannot reproduce: ``` root@labsdb1001[s51306__copyright_p]> SELECT * FROM copyright_diffs ORDER BY diff_timestamp DESC LIMIT 1; +-------+---------... [23:48:48] REMINDER about the planned data center migration: There will be NO EDITING for about 30 minutes on both Wednesday, 19 April at 14:00 UTC and Wednesday, 3 May at 14:00 UTC. This will affect ALL of the wikis. You will be able to read the wikis and to log in, but you will not be able to edit. Read https://meta.wikimedia.org/wiki/Special:MyLanguage/Tech/Server_switch_2017 for more information. Please share this news. [23:48:53] If you find unexpected problems (e.g., can't read for more than a few minutes), then please report those in #wikimedia-tech, which will be monitored by members of the Operations team. #wikimedia-operations will be moderated until the switch is finished and everything is stable again.