[00:11:50] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Research-and-Data, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2727445 (10bd808) I have stubbed out https://meta.wikimedia.org/wiki/Research:Annual_Tool_Labs_Survey/2016 as the location to report the survey result... [00:16:58] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Research-and-Data, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2727452 (10leila) @bd808 * Just to make sure: you will exclude the one user who had asked to be opted out from future emails, per Yuvi? * I read th... [00:40:44] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:44:27] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Research-and-Data, 15User-bd808: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2727473 (10bd808) >>! In T147336#2727452, @leila wrote: > * Just to make sure: you will exclude the one user who had asked to be opted out from future... [05:16:31] PROBLEM - Puppet run on tools-prometheus-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:51:28] RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:01:13] 10Tool-Labs-tools-Pageviews: Add hourly/monthly stats to Siteviews - https://phabricator.wikimedia.org/T148610#2727721 (10MusikAnimal) [07:02:26] 10Tool-Labs-tools-Pageviews: Convert Sass to Less - https://phabricator.wikimedia.org/T142402#2727733 (10MusikAnimal) 05Open>03Invalid Will be using a Node-based Sass compiler in T147966 [08:40:11] 06Labs, 10Beta-Cluster-Infrastructure, 15User-Joe: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2703948 (10Joe) After looking a bit at the status of the cluster, I just decided to recreate all of it from scratch using `role::etcd::common` as a base, via the Horizon UI. [08:42:59] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2727848 (10Marostegui) @chasemp are we talking about this user from labsdb1008 that you want to get replicated to the other labs hosts? If so, I will get... [08:57:51] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2727898 (10jcrespo) A couple of things- I created a new pasword for something similar to this, which is currently living on palladium secrets git store. I... [09:04:31] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2727921 (10Marostegui) @jcrespo thanks - I see it now. You suggest using the same password for this new user? or even using the same user? @chasemp do yo... [11:04:36] (03CR) 10Ricordisamoa: "Actually I wouldn't have merged this one by myself :)" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/316559 (owner: 10Ricordisamoa) [11:16:58] 06Labs, 10Beta-Cluster-Infrastructure, 15User-Joe: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2728207 (10Joe) `deployment-conf03` has been converted to `role::etcd::common` as well, fixing a small puppet glitch along the way. Also, I cleaned up most hiera redundant... [11:17:08] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2728211 (10Joe) [11:17:10] 06Labs, 10Beta-Cluster-Infrastructure, 15User-Joe: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2728210 (10Joe) 05Open>03Resolved [12:56:27] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2728476 (10chasemp) >>! In T148560#2727848, @Marostegui wrote: > @chasemp are we talking about this user from labsdb1008 that you want to get replicated t... [13:08:09] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2728548 (10Marostegui) >>! In T148560#2728476, @chasemp wrote: >>>! In T148560#2727848, @Marostegui wrote: >> @chasemp are we talking about this user from... [13:16:08] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2728584 (10Krenair) I think it needs to be able to at least read non-_p databases to be allowed to create views selecting them? [13:19:42] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2728602 (10chasemp) At the moment the user has full permissions iirc. @Marostegui for my part I meant more like "Let's not use this user for anything els... [13:50:07] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2728703 (10Marostegui) As @chasemp and myself agreed on IRC, we have decided to do some tweaking with a new user (to avoid touching maintain-views one) .... [14:16:08] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2728781 (10chasemp) With maintain-views2 error: > pymysql.err.InternalError: (1227, 'Access denied; you need (at least one of) the SUPER privilege(s) for... [14:41:10] PROBLEM - SSH on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:00] RECOVERY - SSH on tools-webgrid-lighttpd-1201 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [14:51:16] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2728901 (10Marostegui) So far I have only been able to create it with either SUPER or ALL PRIVILEGES ON: `GRANT SELECT ON *.* TO 'maintain-views2'@'10.64.... [14:54:51] 06Labs, 10Beta-Cluster-Infrastructure, 15User-Joe: Remove Labs uses of etcd and confd classes - https://phabricator.wikimedia.org/T147800#2728905 (10Andrew) Thank you! [14:57:42] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2728907 (10Andrew) [14:57:52] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2686048 (10Andrew) 05Open>03Resolved [15:08:28] 10Wikibugs, 06Services (blocked): Figure out how to match phabricator milestone tags for wikibugs reporting - https://phabricator.wikimedia.org/T148650#2728948 (10GWicke) [15:09:02] 10Wikibugs, 06Services (blocked): Figure out how to match phabricator milestone tags for wikibugs reporting - https://phabricator.wikimedia.org/T148650#2728960 (10GWicke) [15:09:57] 10Wikibugs, 06Release-Engineering-Team, 06Services (blocked): Figure out how to match phabricator milestone tags for wikibugs reporting - https://phabricator.wikimedia.org/T148650#2728948 (10GWicke) [16:13:41] 06Labs, 10Tool-Labs: Create more trusty nodes in anticipation of the default for jsub switching to trusty - https://phabricator.wikimedia.org/T147205#2684787 (10bd808) p:05Triage>03High [16:14:04] 06Labs: Increase quota for tools project - https://phabricator.wikimedia.org/T146322#2729091 (10bd808) p:05Normal>03High [16:16:33] !log maps Disabling puppet across maps hosts to unmount nfs shares from labstore1001 (T147657) [16:16:34] T147657: Move maps share to labstore1003 - https://phabricator.wikimedia.org/T147657 [16:16:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL, Master [17:17:25] !log maps Snapshot and syncing latest copy of maps share to labstore1003 (T147657) [17:17:26] T147657: Move maps share to labstore1003 - https://phabricator.wikimedia.org/T147657 [17:17:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL, Master [17:21:00] (03PS1) 10BryanDavis: jsub: Make trusty the default release target for jsub commands [labs/toollabs] - 10https://gerrit.wikimedia.org/r/316823 (https://phabricator.wikimedia.org/T143284) [17:31:13] (03PS2) 10BryanDavis: jsub: Make trusty the default release target for jsub commands [labs/toollabs] - 10https://gerrit.wikimedia.org/r/316823 (https://phabricator.wikimedia.org/T143284) [17:32:04] !log tools.wikibugs Updated channels.yaml to: 56bdfd42cd03c3cb7322e415f7c5db90e07ce60c Update wikimedia-services mapping [17:32:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [17:32:17] 10Wikibugs, 06Release-Engineering-Team, 06Services (blocked): Figure out how to match phabricator milestone tags for wikibugs reporting - https://phabricator.wikimedia.org/T148650#2728948 (10valhallasw) It seems the configuration in git is probably correct (see notes below), but just not deployed. I have man... [17:33:13] yuvipanda, does a k8s webservice have access to anything outside its webroot? [17:33:37] specifically, can a php script run a git pull in /data/project/wikibugs/wikibugs2 ? [17:34:53] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:35:05] PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:35:11] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:36:09] 10Wikibugs, 06Release-Engineering-Team, 06Services (blocked): Figure out how to match phabricator milestone tags for wikibugs reporting - https://phabricator.wikimedia.org/T148650#2729298 (10Pchelolo) 05Open>03Resolved a:03Pchelolo @valhallasw Looks like we've started to get the notifications in the se... [17:37:47] PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:57:44] valhallasw`cloud: yeah [17:57:50] you have access to all of /data/project [17:57:51] so you can [17:58:08] assuming you give the tool users' appropriate permissions [17:58:10] so same as it is now [18:04:13] yuvipanda: oh, but of course git is not installed :D [18:04:33] valhallasw`cloud: yeah, I've been meaning to fix that this week [18:04:41] just need to add a base list of packages... [18:05:32] yuvipanda: ok, no problem. That just means we have to manually pull wikibugs for now. [18:05:37] legoktm: ^ [18:06:36] PROBLEM - Puppet run on tools-exec-1419 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:07:53] PROBLEM - Host tools-exec-1414 is DOWN: CRITICAL - Host Unreachable (10.68.17.216) [18:07:59] PROBLEM - Puppet run on tools-exec-1417 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:14:35] yuvipanda! [18:14:44] addshore! [18:14:48] I'm ready to push that thing out onto production grafana btw [18:15:07] addshore: ah, hmm. can you get a +1 from godog? [18:15:15] if you do I can help push it out [18:15:21] addshore: is there a demo of it in the labs grafana? [18:15:21] * addshore finds a link [18:15:36] yuvipanda: there is / was, let me find / make that link too [18:18:33] So, godog yuvipanda the patch is https://gerrit.wikimedia.org/r/#/c/314029/ and the annotations in https://grafana-labs.wikimedia.org/dashboard/db/revisionsliderenablesdisables come from it [18:19:24] Hmm, but that dashboard only works if you are logged in as an admin ... https://grafana-labs-admin.wikimedia.org/dashboard/db/revisionsliderenablesdisables [18:19:50] 403 to /api/datasources/proxy/3/render .... [18:20:38] hmm [18:20:47] maybe there's specific whitelists in the apache proxy? [18:21:19] but other dashboards work just fine...? https://grafana-labs.wikimedia.org/dashboard/db/eventbus [18:22:20] PROBLEM - Host tools-exec-1418 is DOWN: CRITICAL - Host Unreachable (10.68.17.164) [18:22:51] addshore: are they hitting the new datasource? [18:28:08] No, it is the calls going to production graphite from labs grafana that are failing, but I'm not sure they should be trying to do through the proxy.... [18:28:16] ah [18:28:26] addshore: can you open a bug for it? [18:28:30] probably something something CORS [18:28:50] *opens a bug* [18:31:14] 06Labs, 10Labs-Infrastructure: Proxy calls on Labs Grafana get 403 while not logged in - https://phabricator.wikimedia.org/T148669#2729532 (10Addshore) [18:31:20] yuvipanda: ^^ how does that look? [18:34:32] addshore: looks good. I can take a look later, not right now tho :( poke godog too for prod push :) [18:34:42] will do [18:35:06] RECOVERY - Host tools-exec-1418 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [18:35:26] RECOVERY - Host tools-exec-1414 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [18:38:18] PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:04] RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [18:41:36] RECOVERY - Puppet run on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [18:44:52] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:10] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [18:45:37] PROBLEM - Puppet run on tools-exec-1418 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:46:06] Sorry for being so annoying. I realy need a solution to my problem. Toollabs doen't like jar files!, why? [18:48:01] RECOVERY - Puppet run on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [18:48:53] Every time I try to run jar file, it give me different errors. I've try many ways, but the problem still. [18:50:20] ASammour_: :( java is not very widely used in our tools community. [18:50:35] are you still having issues getting a cron job to run your jar? [18:50:49] Yes, only with cron. [18:51:05] The normal way to run jar files do just fine [18:51:19] But the cron?. No!. [18:52:14] ASammour_: are you having classpath issues or some other problem? [18:52:47] RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [18:53:09] It may be easiest to create a shell script that sets up the classpath, etc and run that from cron rather than trying to cram all of the command line need into a single cron line [18:53:17] Sometimes I get errors like "Error: Unable to access jarfile", but the paths are 100% true. [18:54:36] Also, sometimes I get fatal errors. Notice this errors just appear when using crontb [18:55:24] when things work, are you using jsub to run the jar on the job grid or are you jsut running locally on one of the bastion servers? [18:56:52] One thing that you may need to run a java process is to use '-l h_vmem=...' to raise the default memory limit [18:58:18] RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [18:59:54] No, also using jsub, it does not work. [19:00:23] I just want to know, is there any user who run java bots? [19:00:38] Or it's just me? :) [19:01:12] I know of one java bot that *used* to run. But it has been shutdown. [19:01:23] I'm pretty sure there are at least a few others [19:01:38] but php and python are the vase majority [19:01:40] *vast [19:02:20] If jsub isn't working either then your problem is likely related to classpaths adn/or memory constraints [19:03:07] can you share the command line that you use interactively with me? I might be able to make some educated guesses [19:03:39] Surely. [19:04:27] First I change directory to the right folder, and specifically to the folder which contain jar file. [19:05:06] Second, I typed crontab -e, Then write * * * * * java -jar test.jar [19:05:47] Third, I save the file, and then watch the results in the log file. [19:05:54] Always errors!. [19:06:14] the crontab is not going to capture your current working directory, so you will need to either use absolute paths or make a shell script that does the cd step for you [19:06:55] I've tried write the absolute path, and it does not work either. [19:07:09] so I would first try something like: jsub -once java -jar /path/to/the/test.jar [19:07:14] But the shell script, I didn't try it yet. [19:08:29] testing with jsub will give you largely the same experience that you will have from cron. Your crontab actually needs to use jsub in the command line you provide [19:09:02] The path is like /data/project/sammour/testt/dist/testt.jar [19:09:07] so once you work out a 'jsub ...' command that works you jsut copy it into the crontab and ass the schedule [19:09:27] Do I need to include root directory? [19:09:38] heh. add, not ass [19:09:54] yes, assume nothing about the environment works best [19:10:05] it certainly won't hurt anything [19:10:36] jsub -once java -jar /data/project/sammour/testt/dist/testt.jar [19:10:46] It give me this "Your job 2179514 ("java") has been submitted" [19:11:08] good. now look for 'java.err' and 'java.out' files [19:11:56] you can see if it is still running with the 'qstat' program [19:12:23] I got this error: "Error: Unable to access jarfile /root/data/project/sammour/testt/dist/testt.jar" [19:12:43] hmmm.. where is that /root part coming from? [19:12:57] I type it. [19:13:37] 06Labs, 10Labs-Infrastructure, 06Services (watching): Novaproxy in labs strips out the ETag header if gzip is enabled - https://phabricator.wikimedia.org/T148676#2729694 (10Pchelolo) [19:14:06] I try without root, and it give the same result. [19:14:50] ASammour_: isn't the file actually /data/project/sammour/testt/dist/my.jar ? [19:15:11] there is no /data/project/sammour/testt/dist/testt.jar file on the filesystem [19:16:28] Ah, sorry about that, now it give me this error: "Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit." [19:16:52] Those two errors are always appear to me. [19:17:09] getting closer! I think you may need more RAM to start a JVM. [19:18:03] try this: jsub -once -l h_vmem=1G java -jar /data/project/sammour/testt/dist/my.jar [19:18:16] that will let your process have up to 1G of ram [19:19:51] I got these lines in out log "Error occurred during initialization of VM Could not reserve enough space for object heap Error occurred during initialization of VM Could not reserve enough space for object heap" [19:20:04] closer still! [19:20:23] that is definately telling us that you need more memory [19:20:25] There are no errors. [19:20:55] 10Wikibugs, 10Phabricator, 06Services (blocked): Figure out how to match phabricator milestone tags for wikibugs reporting - https://phabricator.wikimedia.org/T148650#2729737 (10greg) [19:21:05] try this one next: jsub -once -l h_vmem=4000M /usr/bin/java -server -Xms256m -jar /data/project/sammour/testt/dist/my.jar [19:21:44] this is telling the grid that you can have up 4000M of ram and then telling java to try to start with a small initial heap [19:22:25] java loves to have a lot of ram and the job grid is pretty stingy by default [19:23:23] Magnificent! Now it work!. [19:23:38] awesome! [19:24:41] Now, what should I print in cron? [19:25:16] so that same line can just go into your crontab. It would be best not to run it every minute like you were showing me before though. something like: * */5 * * * jsub -once -l h_vmem=4000M /usr/bin/java -server -Xms256m -jar /data/project/sammour/testt/dist/my.jar [19:25:28] that would run the job every 5 mintues [19:26:22] 'man -s 5 crontab' will explain more [19:27:37] we generally don't like things that run more often than once every 5 minutes [19:27:53] just because of the load it puts on the system to start the jobs [19:29:18] Thank you very very much my friend. I really appreciate your help. I just write cron every minute (for testing only). I commonly run my scripts once, or twice a day [19:29:37] perfect. I'm glad I could help you out [19:29:56] Thanks one more time. [19:30:28] It would be great if you made some notes on wikitech somewhere about these tricks for java (full paths to jars, more ram, small initial heap) [19:31:00] maybe here would be a good place -- https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#FAQ [19:31:28] and give some of the errors you saw in case people search for them [19:32:13] * bd808 takes off to run errands and eat food [19:33:23] I promise I will. [19:38:46] PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:45:24] 06Labs, 10Tool-Labs: Create more trusty nodes in anticipation of the default for jsub switching to trusty - https://phabricator.wikimedia.org/T147205#2729818 (10yuvipanda) [19:45:26] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2729819 (10yuvipanda) [19:45:28] 06Labs: Increase quota for tools project - https://phabricator.wikimedia.org/T146322#2729815 (10yuvipanda) 05Open>03Resolved a:03yuvipanda I did it! [19:49:19] PROBLEM - Puppet run on tools-exec-1415 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:52:37] PROBLEM - Puppet run on tools-exec-1419 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:53:47] RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [19:56:03] PROBLEM - Puppet run on tools-exec-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:58:45] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:00:54] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:01:12] PROBLEM - Puppet run on tools-exec-1416 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:04:00] PROBLEM - Puppet run on tools-exec-1417 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:04:18] RECOVERY - Puppet run on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [20:07:36] RECOVERY - Puppet run on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [20:11:03] RECOVERY - Puppet run on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [20:11:13] RECOVERY - Puppet run on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [20:13:47] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [20:15:26] !log tools.heritage Deploy latest from Git master: 9f67e94, c64c22e, a21517e [20:15:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [20:15:53] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [20:17:45] (03CR) 10Jean-Frédéric: [C: 032] Require that orderby parameters are included in requested rows [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315488 (https://phabricator.wikimedia.org/T132644) (owner: 10Lokal Profil) [20:18:41] (03Merged) 10jenkins-bot: Require that orderby parameters are included in requested rows [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/315488 (https://phabricator.wikimedia.org/T132644) (owner: 10Lokal Profil) [20:18:59] RECOVERY - Puppet run on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [20:22:30] PROBLEM - Host tools-exec-1420 is DOWN: CRITICAL - Host Unreachable (10.68.22.103) [20:23:13] 06Labs, 10Tool-Labs: Tools: Migrate puppet roles from ldap to the new puppetbackend - https://phabricator.wikimedia.org/T148683#2729915 (10Andrew) [20:38:27] (03PS3) 10Jean-Frédéric: Mount SQL files via a volume instead of copying in Docker setup [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 [20:39:08] (03CR) 10Jean-Frédéric: "> > > > OK. When I try to follow the steps I get the following error" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [20:40:31] (03CR) 10Jean-Frédéric: Mount SQL files via a volume instead of copying in Docker setup (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [20:42:57] !log tools.heritage Deploy latest from Git master: 213b4a9 (T132644) [20:42:58] T132644: API output is broken when orderby parameters are not requested was (API HTML output is broken when the municipality or address fields are not requested) - https://phabricator.wikimedia.org/T132644 [20:43:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [20:45:23] valhallasw`cloud: ahh, so much effort :( [20:56:14] legoktm: yuvipanda fixed it already! [20:56:37] After a webservice restart the magic pull script works again [20:57:24] yay :D [20:57:28] back to laziness [20:57:47] I'm just realizing — many months late — that it looks like tags are no longer in the revision table on the labs replica database [20:57:58] is that the case? [21:02:24] ragesoss: tags? [21:03:48] https://m.mediawiki.org/wiki/Manual:Revision_table does not list any tag field as far as i can see [21:11:15] @bd808: I've wrote this simple tutorial about running java tools in tool-labs. Hope it will be useful. [21:11:28] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Java [21:11:57] ASammour_: thank you :) [21:14:03] PROBLEM - Host tools-exec-1420 is DOWN: CRITICAL - Host Unreachable (10.68.17.95) [21:24:04] valhallasw`cloud: these: https://en.wikipedia.org/wiki/Wikipedia:Tags [21:25:14] in a separate table, I guess: change_tag [21:28:30] hmm... [21:28:37] I guess it's still there. [21:29:01] not sure what changed ... time to dig deeper. [21:52:42] PROBLEM - Puppet run on tools-docker-builder-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:02:12] (03CR) 10BryanDavis: "the failure from the debian-glue job seems to be just a CI problem. The package builds using pdebuild on tools-docker-builder-02." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/316823 (https://phabricator.wikimedia.org/T143284) (owner: 10BryanDavis) [22:27:41] RECOVERY - Puppet run on tools-docker-builder-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:32] 06Labs, 10Tool-Labs, 10Adminbot, 06Operations: [IDEA] Backup bot for morebots - https://phabricator.wikimedia.org/T148694#2730198 (10Zppix)