Scale: 4 predictions for 2021

Everybody talks about scaling up and out, but what is changing around this in practice heading into 2021?


This is a contributed article by Patrick McFadin, Vice President Developer Relations, DataStax.

Scaling IT systems is important over time - it keeps existing services going, it allows our new applications to grow and serve what users want, and it ensures that companies can grow over time too. From an IT perspective, thinking at scale is still one of the hardest jobs that exists. What worked when you had 100,000 customers per day won’t work when you have millions of customers over the same time period.

Today, with more companies relying on their online services to survive, being able to understand what happens around scaling up is important. There are more and more apps getting created every day - around 111,000 new Android apps and 38,000 new iOS apps get released every month according to Statista, and each one has the potential to grow by millions of users.

So what will happen around scale in 2021?

Prediction #1 - Stateful services in Kubernetes will be in the spotlight

In my predictions for 2020, I thought that Kubernetes Operators would become more important to the speed of Kubernetes adoption. I would say that this was correct to a point. The release of Operators to help integrate Kubernetes within wider application deployments certainly did take place, but this led to some more challenges too.

In the Apache Cassandra community, Kubernetes Operators were a heavy focus last year, but it was clear that there is a lot more to successfully running a stateful service like a database in Kubernetes. Choices in networking and storage can make a significant difference. All of these topics will become the largest discussion in the Kubernetes community. While cloud native applications are now easier to deploy, cloud-native data is still at the beginning - this will need more work on how to integrate Kubernetes into other projects over time.

Prediction #2 - We will care less about databases, and more about data

In the past, you could get kudos for the size of the database that you were administering. Today, that is no longer the case. While the volume of data that we create has gone up, we care more about how easy it is to use that data.

We have databases in place now that can run across cloud services, on-prem or hybrid, and deal with some of the biggest issues around availability and reliability. However, we almost take that for granted - we expect our services to be available around the clock through a browser window, and we are surprised when they are not.

For developers, it’s about reducing the tradeoffs and how easy it is to stand up and run a database in the cloud. The growth of services to support MongoDB, Cassandra and PostgreSQL in the cloud has been massive in response. However, over the next year, we’ll see even less thought going into which database to run and instead what languages and APIs to use instead.

This puts the emphasis firmly on data, rather than databases or infrastructure, and how to work with that data at scale. That infrastructure will still be important. In fact, it will have to be more reliable, more available and more trustworthy than it is at present - but it will be the interest of a smaller group of specialists compared to now. Instead, it will be about making it easier for developers to take and use all this data over time.

Prediction #3 - Multi-cloud will move beyond cost control

Looking at analyst predictions, multi-cloud will continue to grow alongside more use of cloud in general. There are several viewpoints on this. On one side, you have those that see multi-cloud as a waste of resources because you can’t optimise for a given platform and really make the most of it. On the other, you have those that view multi-cloud as an exercise in cost control and keeping cloud companies honest.

The truth is that both sides here are right and wrong all at the same time. Multi-cloud can keep you in control of your destiny, but it’s not the only benefit to this approach. Similarly, making use of cloud providers’ tech should be your goal so that you can wring the best performance out of your cloud instances and spend. From a scale perspective, looking at how to run across multiple cloud services can help with availability, prevent latency and ensure that services meet customer expectations.

In practice, this means looking beyond multi-cloud as a simple “either / or” approach that has one answer. Instead, enterprises will look at mixing and matching multi-cloud deployments to get the best out of them. Where there is a competitive advantage to going with one cloud provider’s approach they will take that; where that advantage comes at a high cost of being tied to a specific approach, this will force more roadmap discussions.

Prediction #4 - Traditional IT team thinking will make a comeback

Over the past few years, it’s been assumed that the smartest minds in IT must work for hyperscalers. The theory here is that those companies are at the cutting edge of technology, so everyone must want to work for them. In turn, this attracts more of the smartest people, and so these companies lead the way.

There is an element of truth here. They run some of the largest and most complex environments on the planet, and they support a huge number of companies in doing their own innovative things. However, they don’t have the monopoly on innovation today, and this will be recognised in 2021.

What the pandemic this year showed is that IT teams can be enormously creative when it comes to problem solving. When faced with huge changes in how they could do business and even survive as going concerns, these teams rolled out initiatives in days. A good example of this is The Home Depot rolling out a complete curbside pickup service from scratch in thirty days.

These teams could only do this by being agile, by moving quickly and by innovating, all the usual terms that we associate with cloud. However, this kind of activity goes on every day within traditional enterprises too, and these teams have now proved they can deliver. In 2021, those initial wins should lead to more trust within the business, and bigger projects in future. At the same time, it will be an opportunity for us all to recognise that disruption can come from anywhere, if we are willing to listen.

These companies are already used to working around scale, and they have made their moves into cloud effectively over the last few years. What will be different in 2021 is how much leeway IT will get to be innovative and follow through on those kinds of projects. For those that were successful in 2020, the next year will be really interesting.