Professional Tips Best Practice GitOps Development
02/17/2023
Software operations and development are combined in the DevOps methodology to create a more dependable and effective software development process. DevOps main goal is to speed up software delivery and deployment while minimizing mistakes and maximizing automation. DevOps has emerged as a crucial element of contemporary software development, allowing teams to accelerate development cycles, deploy software more quickly, and enhance overall quality. This post will look at seven methods for using DevOps to accelerate development.
But what do after you built CI/CD? here are 7 interesting practices that will speed up development and be useful.
A complete replica of the production environment, although on a smaller scale, must be created for each Pull Request, replete with its own domain name, database, and all necessary dependencies. By eliminating the need for the developer to worry about creating their own environment in this scenario, we have a consistent environment and do away with the age-old issue of "working locally."
The engineering working procedure is to outline a set of actions to generate a distinct copy of the namespace in Kubernetes, install the application using a chart (which is naturally maintained in the same repository), and optionally configure with parameters in the Jenkinsfile for producing a Pull Request.
The developer receives a DNS name, a unique namespace, and the capacity to initiate redeployment.
Depending on the current infrastructure (local environment, unique for the entire team || single server for the team that frequently crashes, etc.), accelerating development can have a significant impact.
In 2023, autoscaling should be incorporated into the service throughout the design phase and implemented by default with no exceptions. The service ought to provide simple parallelization capabilities, graceful shutdown, and useful logs. Make sure to set up resource restrictions and requests.
It is necessary to isolate and automatically scale to load each more or less resource-intensive service. For instance, AWS EKS features a Cluster Autoscaler that adds the necessary number of nodes to the Autoscaling Group in the event that the number of pods does not fit in the node pool. quite cozy.
Things become much more interesting when services are automatically scaled: of course, you can do this again using the Horizontal Pod Autoscaler, but the disadvantage of the technique is that it adds 1 pod instance per. Thus, if your load has significantly grown, you will need to wait until HPA adds one pod at a time.
Scaling by events is advised rather than by CPU load plus HPA. And certainly not using HPA + custom metric. For instance, the service performs 10 tasks per second and integrates with Amazon SQS. One million thousand tasks eventually join the queue. Of course, you can wait 27 hours, add HPA, and add 1 pod, but scaling the node pool and making as many pods as you can so that all tasks can be completed in 30 minutes is the best course of action. How? Event-driven autoscaling is KEDA.
Event-driven auto scaling inside Kubernetes.
Engineering best practice: Add the ScaledObject manifest, reference the Amazon SQS queue in the parameter, and set the scale factor in the chart with the application. Name the deployment that will be scaled, if you please.
There are a ton of scalability choices available to KEDA (Prometheus, Redis, Azure, Cloudwatch, etc.), but one that is very intriguing is external. The documentation lists all of the scalers that are available.
The outcome is a microservice architecture where tasks are completed much more quickly, time is saved putting out fires caused by manual scaling, and data is not lost while processing SIGTERM when containers are treated like a herd (riding herd over your microservices) instead of a single point of failure.
Side effect: In the case of static deployment or manual scaling, we don't waste time on pointless searches for flaws and addressing them because we do it correctly from the start.
Scaling and rate restrictions work together to prevent the application from overload. The system will always have a bottleneck, according to the Theory of Constraints, and our duty is to identify it. It's a database, sluggish cold object storage, or anything else quite frequently. Our job is to scale the application to this limit and return 429 Too Many Requests to the users. Giving a 429 error is usually preferable to allowing the application to crash or, in the case of a database, to operate much more slowly.
The drawback is that clients must implement the Circuit Breaker pattern, which causes timeouts between requests to an overloaded system to climb exponentially.
Istio should be added to the stack, Local Rate Limit should be used, and sidecar in the pod should be used. As a side effect, Istio provides us with superb real-time analytics for status codes, latency, and future tracing.
The service is then protected from overload, and developers are spared the time-consuming task of looking for problems along the lines of "a lot of traffic came in" or "failed to handle the load." Second, we can plan more precisely since we are aware of the system's constraints.
You lose when machine learning trials take months while rivals take just hours. Training should not take too long because data is the new gold. With distributed train, you can run workloads across a large number of servers while KubeFlow scales and manages pipelines.
The most recent KubeCon + CloudNativeCon shown that running ML workloads on KubeFlow is the best option. Although installation and configuration are quite difficult (how much does it cost to construct kfctl + ArgoCD), the outcome is worthwhile.
An ML engineer can save hundreds of hours with this solution.
A practice that will aid in the implementation of a dynamic development environment or be beneficial when developers are active in infrastructure is bundling application dependencies with it. Particularly appropriate for automating client normal chores - I created a manifest that works for everyone.
Depending on the cloud provider, you must choose GitOps for the infrastructure in order to apply this technique. At Amazon, we wouldn't yet dare to utilize AWS Controllers for Kubernetes, however there is a fantastic implementation from CNCF - CrossPlane - that works perfectly for GCP.
The key is being able to deploy the application along with small bits of infrastructure (S3 buckets, SQS queues, see supported resources) that can be described in YAML. The controller will adopt the modifications and build the required infrastructure. Deletion also functions; on the "PR merge" event, namespace, infrastructure manifests, and these resources are destroyed by the controller.
For dynamic environments, it is standard engineering practice to add minor bits to the helm chart or allow your clients to develop the infrastructure components themselves. When the client makes the modification, it is evaluated and a temporary S3 bucket is established.
The developer can therefore resolve his issue with a regular PR rather than having to wait for DevOps to take up the chore of changing the infrastructure.
DevOps is all about communication, thus we make as many talks and gatherings as we can to explain how the infrastructure and monitoring function, what benefits you can receive from them, and how they may be helpful.
The better, the more openness. The better an application is tailored and the more cutting-edge engineering approaches are used, the more likely it is that the developer will achieve market leadership through competitive advantage.
Organize a number of "How Infrastructure Works" events for anyone who might be interested and see how the suggested strategies are increasingly being used. Dropping a link to Grafana is one thing; demonstrating how to connect the Prometheus SDK to an application, create a graph, or create a Slack alert is quite another.
The monitor should be the sole one to report issues with the system. You have no tracking whether customers contact you (or write to support) and use your system. The appropriate response should be "Thank you, we know, we got an alert and answered in the chat, we're resolving it." Even if this is your QA or a developer who just saw a 503.
Kube-Prometheus-Stack components.
Helm install prometheus-community/kube-prometheus-stack is the simplest possible choice. You will receive a basic set of metrics, graphs, and warnings that will cover the 80% of issues in the Kubernetes cluster as well as the most common cases (disk clogged, restarted, task killed).
Long-term storage should also be included, metrics on apps should be described in the OTEL (OpenTelemetry) format, and infrastructure as a code should be used for alerts and displays.
As a result, we will have a monitoring system that can anticipate potential issues, explain how the system is currently functioning, and dramatically save debugging time. Not to mention preserving nerves, cash, and a public image in the event of production-related interruptions
In real-world settings, a DevOps team's contributions go beyond CI/CD alone and can have a significant impact on a business. Share this post with your DevOps team or bring it to the architect's notice if they appear to be lagging behind.
Any team and organization can speed up the development pipeline, increase product quality, and achieve a Time-to-market improvement ranging from 1% to an indefinite percentage by putting at least one of the techniques from this article into practice.
In Apprecode we already use the latest version of Kubernetes for our leading customers. Please contact us for more information.