MLOPS and JENKINS INTEGRATION

I created a project which shows a huge integration of machine leraning and jenkins . task details are like diffrenrt jobs to complete it. so task discription is -

1. Create a container that has the python3 , tensorflow, pandas and keras installed using the Dockerfile.
2. When we launch this image, it should automatically starts training the model in the container.
3. Create a job chain of JOB1, JOB2, JOB3, JOB4 ,JOB5 using the build pipeline plugin in the Jenkins.
4. JOB1: Pull the Github repository automatically when some developers push repository to the Github.
5. JOB2: By looking at the code or program file, jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training ( eg. if code uses CNN , then jenkins should automatically start the container that has already installed all the software required for the processing).
6. JOB3: Train your model and predict accuracy or metrics.
7. JOB4: If metrics accuracy is less than 80% , then tweak the Machine Learning model architecture.
8. JOB5: Retrain the model or notify that the model has been successfully created with the required accuracy.
9. Create one extra job JOB6 for monitor: If container where the model is running fails due to any reason then this job will automatically start the container again from where last trained model left.

let’s start

first of all we have to create a machine learning model for recognizing images for this i have created a CNN model . in this model i add different CRP layers and train the model .

Now the next thing is to create a Dockerfile that have tensorflow , keras , numpy and python3 preinstalled. this Dockerfile is something like that when launching any container , that container will automatically start training the model.

Dockerfile

building image:

docker build -t me:8 . (workfolder)

images i created :

image — me:8

job1:

In this job we have to copy all the code whenever any developer push it to github.

job2:

job2 will start if and only of job1 is done successfully . in this job jenkins should automatically start the respective container according to the code written by developer. if this code is of CNN then it will deploy the container that have all the software installed for using the process. for this we have to grep some keyword in the code like i grep keras because it is very usefull in CNN.

launching the container according the code

epochs

job3:

this job will trigger after the job2 comlpeted successfully and predict the accuracy.

accuracy

job4:

in this job we have to measure the accuracy . if the accuracy is less than 80 then it should automatically tweak the code and again train the model and find again accuracy. now if it also not grater than 80 then again tweak . for this they automatically add layer and epochs also and get perfect accuracy.

tweaking the model

job5:

this will trigger after job4 . now if accuracy is greater than 80 then is will mail a message to notify that model is trained successfully.

job6:

job6 will work as a monitering tool . it will moniter all of the previous job . if accuracy is not obtained then it will trigger job2 again. and if the container goes down it will again restart the container.

moniterin all above

GITHUB LINK FOR CODE AND DATASET:

hirendrakumar/mlops1