I watched a video recently about how Scala took over the big data world
I think you mean Spark? I use Spark heavily at work and know little Scala (although I do have an engineering team who do work in it sometimes).
I would add Apache Spark to the list. I watched a video recently about how Scala took over the big data world (probably not true) [1], but the presenter made an interesting point about how Spark subsumes a lot of different things (streaming, machine learning, built in support for SQL) and it is good enough at those things even if not the best tool. Not surprisingly, that actually makes it a good candidate for adoption in the enterprise.
So Spark is great, but it isn't the only thing. For example, it does let people use Python and R on the same platform pretty easily, and with the potential for good performance.
However, you really need to know what you are doing to get the best of it (what a surprise, hey!). For example, Databricks likes to show how Dataframes/Datasets give huge performance advantages over the old RDD programming model.
This is true, but you need to understand why to make sure you see the same benefits. Basically, there are numerous primitive functions than have been implemented as native operations of the Dataframe classes, and if you use them they perform well. If however you want to use Python UDFs, then you won't.
I think you mean Spark? I use Spark heavily at work and know little Scala (although I do have an engineering team who do work in it sometimes).
I would add Apache Spark to the list. I watched a video recently about how Scala took over the big data world (probably not true) [1], but the presenter made an interesting point about how Spark subsumes a lot of different things (streaming, machine learning, built in support for SQL) and it is good enough at those things even if not the best tool. Not surprisingly, that actually makes it a good candidate for adoption in the enterprise.
So Spark is great, but it isn't the only thing. For example, it does let people use Python and R on the same platform pretty easily, and with the potential for good performance.
However, you really need to know what you are doing to get the best of it (what a surprise, hey!). For example, Databricks likes to show how Dataframes/Datasets give huge performance advantages over the old RDD programming model.
This is true, but you need to understand why to make sure you see the same benefits. Basically, there are numerous primitive functions than have been implemented as native operations of the Dataframe classes, and if you use them they perform well. If however you want to use Python UDFs, then you won't.