Innovation and Open Source

we innovate with Open Source to create
awesome applications for you

News 

TupiLabs Report: 20 Jul, 26 Jul

This is a weekly report that we publish every week, with news from 20-Jul to 26-Jul.

We are working for you

Have a great week! :-)

More News

Blog 

Durable Tasks in Jenkins

Today’s post is a documentation that I am writing while I learn how the Durable Task Plug-in for Jenkins works. The durable-task-plugin is not really a plug-in. It is a “library offering an extension point for processes which can run outside of Jenkins yet be monitored”.

Durable Task Plugin

But how does it monitor a process running outside of Jenkins? Well, let’s examine its source code.

TL;DR: This library provides a high level API for controlling long running tasks, and provides shell and windows examples that executes something like nohup sh -c script.sh > log.txt 2>&1; echo $? > result.txt. Appends the result of log.txt to another OutputStream and returns the exit code.

DurableTask

The DurableTask is an extension point, and represents a “task which may be run asynchronously on a build node and withstand disconnection of the slave agent”.

DurableTask Class

The only interesting method is launch(…). This method is called to launch our durable task (whichever it may be). And it receives environment variables, the reference for a workspace, a launcher and a listener (logger).

DurableTask Couplings

Looking at its relationships, we can see that FileMonitoringTask extends it, and that it utilizes two other classes, and is utilized by one class (DurableTaskDescriptor for generics).

The launch(…) method returns a Controller object, our next topic.

Controller

Once your durable task has been submitted, the Controller returned by the DurableTask#launch(…) method will define how to control its execution. The name of methods in this abstract class are very intuitive.

Controller Class

Let’s consider your task is running and it will take a while to complete. The controller provides the writeLog(…) method that is responsible for verifying if your task has produced any new output and keeping track of it.

The exitStatus(…) can be used to retrieve the exit status. But be wary as it can return null if the process is still running.

The stop(…) tries to stop the running task, and cleanup(…) is called afterwards to close any open resources.

Controller Couplings

In the library the controller is only used in the abstract DurableTask and in the sample FileMonitoringTask.

FileMonitoringTask

“A task which forks some external command and then waits for log and status files to be updated/created”. That’s the description of the FileMonitoringTask. Good use cases include those long running shell scripts that output some data for each folder/file processed, or that script that executes Python, R, lua and what not and tries to log to a text file what is going on. Let’s take a look at some of its methods.

FileMonitoringTask Class

In FileMonitorTask, the launch(…) method only adds a special environment variable (which scope it out of the context of this post) and calls the doLaunch(…) method. The doLaunch(…) method is abstract and returns an instance of FileMonitoringController.

FileMonitoringTask Couplings

Looking at the relationships of the FileMonitoringTask we can confirm that it extends DurableTask, and that it utilizes the Controller class. And we can also see that there are two implementations in the durable-task-plugin: WindowsBatchScript and BourneShellScript.

But let’s take a look first at the FileMonitoringController returned by the FileMonitoringTask#doLauncher method, and then conclude looking at the BourneShellScript.

FileMonitoringController

The FileMonitoringController is a static inner class of FileMonitoringTask, and is returned by FileMonitoringTask#doLaunch(…).

FileMonitoringController Class

The constructor of the FileMonitoringController calls controlDir(…). This method is responsible for creating a working directory for this controller. The directory is created within the workspace and its name is a random string (id).

The writeLog(…) method when called to retrieve the task log file first will call getLogFile(…). The latter will create/get a file named “jenkins-log.txt” in the workspace. Having the log file, the writeLog method continues appending the output of the task into this log file, and stores its last location (lastLocation). When invoked again, writeLog(…) will use the lastLocation variable to check if the log changed or not.

Finally, the exitStatus(…) relies on getResultLog(…) to retrieve $WORKSPACE/<control-dir>/jenkins-result.txt. If the file exists, the exitStatus(…) method will return the exit status written in this file, otherwise it will return null.

BourneShellScript

The BourneShellScript extends the FileMonitoringTask. And what is does is incredibly simple:

nohup sh -c '/var/lib/jenkins/.../script.sh' > jenkins-log.txt 2>&amp;1; echo $? > jenkins-result.txt

This can be translated as: run script.sh, which will output to jenkins-log.txt, and after it is completed, echo the exit code to jenkins-result.txt. Now go back and read about the FileMonitoringTask again…

BourneShellScript Class

I told you BourneShellScript was simple. Its constructor receives a script String parameter that contains the script body (returned by getScript()). The doLaunch(…) method is responsible for all the boilerplace code to execute the aforementioned nohup -sh script and redirect the output streams to the right files.

Your long running task will feed the jenkins-log.txt, and once it is completed the BourneShellScript will know because the jenkins-result.txt will exist, and hopefully it will contain the exit code of your task.

NICE

LabLua aceito no GSoC 2014

O LabLua, laboratório de pesquisa da PUC-RIO, foi aceito no Google Summer of Code 2014. Isso quer dizer que estudantes podem se candidatar para trabalhar nas ideias propostas. Os alunos selecionados podem receber uma remuneração total de até USD 5.500 (aproximadamente R$ 12.900,00).

O LabLua é o único projeto Open Source brasileiro na lista de aprovados. Se você é estudante e tem interesse em programar em Lua esta é uma ótima oportunidade.

LabLua

Configurando o Jenkins para usar o Active Directory

Depois de ver algumas mensagens na lista do Jenkins em Português decidir escrever este post para ajudar quem quiser configurar o Jenkins para usar o Microsoft Active Directory.

O Jenkins já possui autenticação e uma base de usuários própria. O que faremos é apenas utilizar o Active Directory (AD) como a base de usuários. Para isso basta instalar o Jenkins Active Directory plugin e reiniciar.

Plug-in do AD

Agora precisamos configurar o Jenkins para acessar o Active Directory. Vá até Gerenciar Jenkins / Configurar Segurança Global e em Controle de Acesso, selecione "Active Directory". Entre o domínio e os dados para se conectar no AD. Caso você não saiba os dados do seu AD, pergunte para alguém de infraestrutura ou tente se conectar ao AD com algum cliente de LDAP.

Configurando o Plug-in

Pronto! Agora basta você escolher o modelo de autorização. Na maioria das vezes utilizo a estratégia baseada em matriz por projeto. Se você utilizar a mesma, adicione o nome de usuário de rede e atribua as permissões corretas.

Espero que ajude, boa diversão!

More Posts

Ideas 

Add tags to Jenkins builds or export specific builds

This idea is from @abayer, from these following tweets:

Tweet 1

And:

Tweet 2

We could simply add tags to builds, and then create some place that would list the builds (or other things) that are tagged appropriately. Or we could add a way to define whicih builds should be exported to some tool, and then via the plug-in API maybe, post to this external tool.

Add support to Promoted Builds on Build Pipeline Plug-in

From https://code.google.com/p/build-pipeline-plugin/issues/detail?id=80

Jenkins Build Pipeline Plug-in, by Centrum Systems (Australia), is an excellent plug-in, used by many users (over 6000 downloads according to last report).

However, it does not work well with the Jenkins Promoted Builds Plug-in, another great plug-in (over 4100 downloads last month).

It means that users are not able to create pipelines based on promotion (promoted pipelines?). This contribution would be awesome, useful and help a lot of users (maybe over 10000? :-)

Enhance MySQL management in Stacks (University of Oregon)

From https://groups.google.com/forum/?fromgroups=#!topic/stacks-users/2HwY2dDyeSM.

Stacks processes RAD DNA sequences and produces output, that is displayed on the Web with a bundled PHP+MySQL application. This application is quite useful, especially since it helps developers to analyze large amounts of data.

stacks_logo

One limitation though, is that some management tasks may need manual execution of SQL’s in the database. Someone could add few scripts or new features to the Web interface (like a management section?). This could help researchers on their research.

More Ideas