Next Generation of CI/CD
Does it really time to adopt new process of CI/CD?
As we all know that today’s in Devops, Jenkins is most widely using in the organization. Basically, it make the entire process like Continuous Integration, Continuous Development, Continuous Deployment automated which help to release the product on target. It looks like we are good with Jenkins.
But why are we experimenting with another CI / CD tool when we are good with Jenkins?
Because of Jenkins extending the native functionality is done through their plugins
It has been observe that sometimes you have to dive into your java code and you figure out that what the hell that some any plugin actually does. And due to this thing are getting so messy.
Jenkins Plugins are expensive to maintain, secure and upgrade. It sometime needs to restart the Jenkins server which one of the most pain-full decision if user is active and jobs are also running.
It also observed if Jenkins salve node which agent. Could stop due to serval reasons then also it very time consuming to reach to that system and start the agent service. And if in case their 200 plus nodes so, certainly you get to know that all agent services are stop working. And that will become blunder.
More over Jenkins become so slow sometime and difficult to fix.
Even other CI tools like TeamCity one major that if of UI. The new user easily lost in UI.
Looking into these issues, yes it really time to come to adopt new process of CI/CD that we have look into for future CI tools. Which can over come with all these drawbacks.
My client also wants to adopt some other CI tool which has security and consistent performance.
After day to day analysis and created some Research and development lab. I have found one of the best tools which will reliable, good performance and including healthy security.
The Name of Next Genration CI which most of the development, and QA companies are adopted is know as
With this CI tool you can perform or achive the entire software development lifecycle and also it contains performance metrix status for every pipeline which has been run. This make a lot difference with other CI tolls. The has major benefit is that monitoring of resource of your application deployed either the application deployed on staging or production on every checkin code. This help to develop project planning and source code management to CI/CD, monitoring, and security.
GitLab is a github like service that organizations can use to provide internal management of git repositories. It is a self hosted Git-repository management system that keeps the user code private and can easily deploy the changes of the code.
This CI tool has in-build feature or you can say main feature it have Continuous Integration/ Continuous Delivery / Continuous Deployment. It enables teams to build, test and release software at a faster rate.
How Gitlab Works?
GitLab is an open-source ‘clone’ of GitHub. It’s functionality is very similar to that of GitHub, but because it is open-source it means that we can use it on our servers for free.
GitLab allows us to create and modify multiple Git repositories online. It allows admins to manage each repository, the users, and their groups. It has a simple user interface that is very easy to use. To kick start on gitlab you create account on GitLab and click the green New project button or use the plus icon in the navigation bar. This opens the New project page. On the New project page you can choose last tab i.e “CI/CD for external repo” and connect your git repositories from git Repo by URL.
CI/CD with Gitlab
GitLab offers a continuous integration service. If you add a
.gitlab-ci.yml file to
the root directory of your repository, and configure your GitLab project to use
then each commit or push triggers your CI pipeline.
tells the GitLab runner what to do. By default it runs a pipeline with three stages:
deploy. You don’t
need to use all three stages; stages with no jobs are simply ignored.
If everything runs OK (no non-zero return values), you’ll get a nice green checkmark associated with the commit. This makes it easy to see whether a commit caused any of the tests to fail before you even look at the code.
Most projects use GitLab’s CI service to run the test suite so that developers get immediate feedback if they broke something.
There’s a growing trend to use continuous delivery and continuous deployment to automatically deploy tested code to staging and production environments.
So in brief, the steps needed to have a working CI can be summed up to:
.gitlab-ci.ymlto the root directory of your repository
- Configure a Runner
From there on, on every push to your Git repository, the Runner will automagically start the pipeline and the pipeline will appear under the project’s Pipelines page.
is where you configure what CI does with your project. It lives in the root of
On any push
to your repository, GitLab will look for the
and start jobs on Runnersaccording
to the contents of the file, for that commit.
.gitlab-ci.yml is in
the repository and is version controlled, old versions still build
successfully, forks can easily make use of CI, branches can have different
pipelines and jobs, and you have a single source of truth for CI. You can read
more about the reasons why we are using
.gitlab-ci.yml is a YAML file so you have to pay extra attention to indentation. Always use spaces, not tabs.
GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI, the open-source continuous integration service included with GitLab that coordinates the jobs
GitLab Runner is written in Go and can be run as a single binary, no language specific requirements are needed.
It is designed to run on the GNU/Linux, macOS, and Windows operating systems. Other operating systems will probably work as long as you can compile a Go binary on them.
If you want to use Docker
make sure that you have version
v1.5.0 at least installed
- GitLab Runner Allows to run:
- Multiple jobs concurrently.
- Use multiple tokens with multiple server (even per-project).
- Limit number of concurrent jobs per-token.
- Jobs can be run:
- Using Docker containers.
- Using Docker containers and executing job over SSH.
- Using Docker containers with autoscaling on different clouds and virtualization hypervisors.
- Connecting to remote SSH server.
- Is written in Go and distributed as single binary without any other requirements.
- Supports Bash, Windows Batch, and Windows PowerShell.
- Works on GNU/Linux, macOS, and Windows (pretty much anywhere you can run Docker).
- Allows customization of the job running environment.
- Automatic configuration reload without restart.
- Easy to use setup with support for Docker, Docker-SSH, Parallels, or SSH running environments.
- Enables caching of Docker containers.
- Easy installation as a service for GNU/Linux, macOS, and Windows.
- Embedded Prometheus metrics HTTP server
Let’s take an Example with demo project on web blog application.
This demo project was made using Spring Boot, Spring Security, Spring Data JPA, Spring Data REST, Docker and Kubernetes. Database is in memory H2.
There is login and registration functionality included.
Now I need to setup CI/CD pipeline in gitlab which will become quite simpler and faster approach to have CI/CD quickly, do this will import the project from github just like you have seen on above screen shot.
After that I have just create
.gitlab-ci.yml just on the root
of repository which I have recently
imported. The yml file will have four jobs in pipeline
- Build = which compliation of build push to artifact. I am using the jfrog artifact.
- Unit Test = perform unit test
- Build-Push Docker image = This job will build docker image , it will copying the build from jfrog and get’s rest of the environment , build an image and push to docker hub repo.
- Deploy= this will deploy the docker images on my staging environment by using kubernetes.
- You need jfrog account or
any artifact or even you can use gitlab own artifact.
- Docker hub account where we can push the docker images
- Working Kubernetes cluster.
Here how the architecture look like
When I will any changes in my repo and commit, this will automatically kick start my pipeline CI/CD jobs, it will first check the gitlab.ci.yml file and will start the CI/CD pipeline jobs. The jobs which are mention in the yml file. And that’s it within few minute I have setup CI/CD and kick off job without waiting time in installing plugin , creating node , installing agent on remote system etc.
The gitlab CI will lunch docker from gitlab runner. Inside that docker it will clone your repository and execute the job.
So, for every jobs in docker will lunch will repo inside that docker and start that job.
Once the job will complete it will look like this, when you will navigate to CI/CD pipeline in your gitlab account.
when job was success you will see green passed, if any of the job will fail it will show in red failed. As we can see in above screen shot.
As we can see on a single dashboard, we can see commit message, pipeline job, and git PR number which makes lot different to other CI tools.
If want to look into the console on each jobs, then I will click on the jobs and their we can see the console logs
Similarly, we can see all other jobs with unit test, docker image build push and deployment via kubernetes.
Integration of kubernetes with gitlab is also very simple here. Only you need to get pull of the information about api url, Service token and CA certificate that’s it when you have these details from your kubernetes cluster you can integrate kubernetes with gitlab.
In my demo project I am deploying the docker images through kubernetes. I have already created kubernetes cluster on AWS
You can create your own kubernetes cluster on your choice
You can create in
etc. where ever you have existing cluster or new cluster you are comfortable to create.
Kubernetes integration with gitlab shown below.
Now, suppose that I need to take a look into my kubernetes pods logs
So if we are using CI tools like jenkins, teamcity then we must create script to pull that information of pod logs.
But if this pod logs can be getting in simple manner or in the CI tool itself.
Is it sound good?
Yes, we like to have any such CI tool in which its own dashboard can show the logs as well also help to track the deployment progress.
And such additional feature only Gitlab provide which I found. These features will be going to be the part of next generation of CI/CD tool which can show, get information of your deployment including performance Metrics under one roof i.e on same gitlab account where you have repository.
Performance metrics which show pods cpu, RAM utilization . It also send the alert when you set thrashhold on your slack or email where ever you want.
Alert & Email Notification
In my demo project on metrics I have created alert that if my any app pod goes above my mention threshold, then it should send me email notification as well.
Deployment Status on Slack #Channel
As we know that Slack is widely used by DevOps and development teams to communicate status. Typically, when a build has been tested and is ready to be promoted to a staging environment, a QA engineer or DevOps engineer kicks off the deployment. Using Slack in a ChatOps collaboration model, the promotion can be done in a single click from a Slack channel. And because the promotion happens through a Slack channel, the whole development team knows what’s happening without checking email.
Now a days taking care of promoting the status of build, it is also important to integrate with your ChatOps, we do with Jenkins but for that we need to install extra plugins and focus on scripting as well to get accurate message.
So, to avoid all the complication we will start thinking some other tool also which open source as well easy to integrate and not soo much dependent.
All this use case fit in gitlab. The Slack Notifications Service allows your GitLab project to send events (e.g. issue created) to your existing Slack team as notifications.
To enable GitLab’s service for your Slack team:
to your project’s Settings > Integration > Slack application (only
visible on GitLab.com)
- Click the “Add to Slack” button
That’s all! You can now start using the Slack
As we can see on above screen shot, I have integrated the slack with gitlab. And when every I kick of the pipeline. It will show me the status about the pipeline.
Finally, as a writer of this blog and worked on gitlab. I could say that GitLab CI has a greater potential because of the fixed integration into the git repository manager Gitlab.
The GitLab CI with its larger ability builds the CI/CD process faster, reliable, and secure,
Now a days many development companies are moving their application architecture from monolithic to microservices. And it has been seen that in Jenkins like CI tools becoming weaker or lengthy, if to develop CI/CD Pipeline whereas GitLab has smooth integration with docker and kubernetes which make this CI tool more relevant for microservices application.
GitLab with its simpler and advantage features like: –
- Easy integration of existing Kubernetes clusters
- Support for multiple Kubernetes clusters
- Easy Deployment of Helm, Ingress, and Prometheus on Kubernetes
- Canary Deployments
- View Kubernetes pod logs
- Re-deploy the current version, or even rollback an old stable one in case something went wrong.
- Browsable artifacts where you can upload your job artifacts in GitLab itself without the need of an external service. Because of this, artifacts are also browsable through GitLab’s web interface.
- Container debugging with an integrated web terminal
- Application performance monitoring
- Application performance alerts
We can say these features in future going to become serious necessary, because if the application becoming growing, we need to track everything like deployment, unit test, smoke test, regression test, acceptance test, performance monitoring, alerts, roll back etc. Then we will need to start thinking for something better approach if choosing CI, which can show everything on same roof and gitlab day by day proving that they will become the future of Continuous Integration, Continuous Development, Continuous Deployment and be the part of Next Generation of CI/CD in DevOps.
In conclusion, we at 360logica choose the CI tools according to client’s requirement. We do Research and Development on different-different trending Devops Tool and share our views, do practical’s on working idea through blog post.
You can also check out Video on Demo of Next Generation of CI/CD in DevOps.