Based on a diverse experience in DevOps complex projects, Viktar, Systems Engineering Team Leader from Belarus, overviews 7 tips on how to encourage the team for changes and growth, as well as how to keep up to date with the innovative tech world (narration is original).
Many people still think that a DevOps Engineer is a kind of a super systems administrator that knows more than other systems administrators. In fact, DevOps — Development Operations — is a culture of development processes. It covers everything: how you set priorities, what technologies you use, how you interact with the client, and more.
The goal of an effective DevOps is to deliver a product the client needs as quickly as possible.
I’ve been working at EPAM for almost ten years now, and I want to share the principles and approaches that we adhere to in the development process.
“To say, “the more the deployment, the better” is like saying “the more lines of code, the better.”
Currently, the whole world uses Continuous Delivery, an approach where software is created in short iterations. But engineers often forget that business also needs to be involved in the process. CD won’t fulfill its potential if you don’t have constant feedback: is the product working the way the client needs? Are people using the new functionality? Does it help to solve business problems?
We always let our clients know how often we’ll do product demos. Even if we only change ‘the color of one button,’ we’ll show it to them at the next meeting and explain our decision. This approach helps us to find integration problems earlier and fix them right during the development phase.
The earlier you find an error, the less time you’ll spend fixing it. This means saving resources – employee hours, server capacity, and so on.
For the client to give you the feedback you want, the new functionality needs to work well. Which brings us to our next principle.
What’s the use of frequent deployments if the product isn’t stable? Our second principle is a successful delivery. If during or after deployment, something isn’t working or is working poorly, the client will certainly have questions. Successful delivery directly impacts the customer's trust.
We recently did a nighttime deployment for a client, but we didn’t take the service down for it. Instead, we limited its functionality and, for any potential nighttime site visitors, displayed a message: “You can view all the content, but for now, you implement any edits, as we’re uploading a new version.” This reduced negative reactions from users to zero.
As for approaches, I’ll mention blue-green deployment. On one of my projects, this reduced both the number of errors and the delivery time (from four hours to thirty minutes).
Blue-green deployment is the creation of a new version of the product that resides next to the old version on the same server. When you’re ready, all you need to do is to switch the client from the old version to the new one. If something in the new version isn’t working, you just switch them back.
Another approach is the canary deployment. Here, we put 5% of the load on the new version and 95% on the old one. Then, we see how users work with the new functionality and how the system handles the load. Earlier, you could do the canary deployment only on expensive equipment, but now anyone can afford it.
People can’t multitask.
Usually what happens is that a developer commits some code and moves on to the next task; now they’ll need time to relearn the details of the first task if there’s an issue.
But if I commit something and immediately see a notification that something’s not working, then I still have the context I need to fix it.
For this reason, we need to give fast feedback – ideally, within seconds. Of course, that won’t always be possible. We try to do it within five to ten minutes so that the developer doesn’t have time to switch their focus to another task.
To do this, we use automation, specifically automated tests. If there are too many tests, we decide on whether to make them parallel or break them into smaller parts, and whether to give test results as they come or in groups. We then save the best practices in our internal project documentation.
You’ve probably encountered teams that think DevOps culture presupposes automating everything – particularly those junior engineers with fire in their eyes. Our developers are no exception, but the decision to automate is made by our tech lead, who calculates the ROI for automation.
Any expense has to be worth it. If I need 160 hours to automate something that we usually spend five minutes a month on, then it’s better to keep doing it manually.
Of course, it’s not always just about ROI. If a task is large, complicated, and hard to repeat, automation can serve as a kind of documentation for it.
There’s also automation of work with cloud providers. If we see that the project is large and long-term, there’s a sense in automating it. But if the client needs an MVP quickly to test the demand, we can do some work manually.
We don’t use technologies for their own sake, but we always tell the client if there’s something we can improve significantly.
First, the client needs to recognize the problem. As an expert in this area, I see the problem and communicate it to the client, then propose a solution and зpresent arguments for my approach.
Sometimes, the client isn’t ready to invest in a new solution and asks us to leave things as they are. In that case, we can still implement technology, only in our free time. Certain technologies make our work easier, so it’s worth it.
At the beginning of my career, I worked with a client that had an ineffective monitoring system. We proposed implementing Zabbix, but the client didn’t want to spend the resources on it. We gradually introduced the technology in various parts of the project on our own just because it was convenient. Eventually, the client saw the advantages and asked us to implement Zabbix on the whole platform.
There’s always a delivery team working on a product. This team is responsible for the final delivery of the product, and not just for some fragment of the work. We didn’t invent the term delivery manager - it was already in use on the market.
In practice, EPAM delivery managers are technical people, usually former developers. Even if they don’t write code now, they are capable of conducting a code review.
They know how applications, platforms, and components work. They can coordinate architecture and technology decisions. This doesn’t mean that they’re doing all these roles at once, but they know the basics and can organize a team.
Delivery teams are combinations of people with the right skills. There could be a business analyst, QA tester, UX designer, software developers with the needed stack, and so on. Each team is set up individually depending on the project, and it’s the delivery manager who sets them up.
As a rule, delivery managers specialize in a single industry, like finance or eCommerce, and keep up with industry trends. This is what makes delivery managers so valuable to the client: they understand the market and can propose solutions that are good for business – even if this wasn’t an original task.
DevOps is a culture, and that culture determines how effectively development processes are set up in a company – and how valuable those processes will be for the company, client, and user. We add all best practices to our project documentation and share them with our colleagues.
Don’t undervalue social learning — when you study with someone, not alone – it really helps. For one, it’s motivation not to quit; for another, you encounter other viewpoints and approaches to the same problems. Right now, I’m preparing for the Certified Kubernetes Administrator exam, along with my colleagues. We meet twice a week in the evening and look at different cases. The exam program is great – you immediately see how you’ll use the shared information in practice.
A lot of specialists are still not sure about where to start learning DevOps. I think you should start with the basics, the fundamental knowledge that young professionals might not always possess.
To really master DevOps, of course, it’s not enough just to learn the principles, learn a technology and remember a list of services. You have to work in a company that truly follows those principles.
Whatever they say about containers, understanding the core of operating systems is still important in 2020. Windows or Linux — your choice.
Python, Bash, Powershell. You have to be able to automate routine tasks.
I recommend starting with one: Azure, AWS, or Google Cloud. Then, you can use CI/CD tools like Jenkins. Then, take a look at containers: Docker and Kubernetes.