Mojo vs Spark and Julia

[hmcbride]

How does Mojo differ from Apache Spark , i.e. will it be suitable for the large scale data processing etl pipelines that Spark is currently used for. Is it suitable a scalable scientific computing language like Julia, if the core of Mojo is the ability to do Matrix operations at speed and scale , could you use it to replace Matlab , as in essence Matlab (Matrix Laboratory) is just a bunch of Linear Algebra routines

[mojodojodev]

Apache Spark is a very different thing to a programming language, it's basically an API for distributed computing with a huge amount of complexity to get it working if you're self hosting, while cloud providers have managed versions that look after all the complexity for you. It would be more comparable to the Modular Inference Engine which will allow distributed computing on machine learning models via an API. Mojo is a programming language that they built specifically to make the kernels for that Modular Inference Engine in a way that can be reused across CPU's/GPU's/TPU's etc by lowering to MLIR. They're still using C++ to coordinate machines for the distributed computing, and are adding features to Mojo so they can replace C++ throughout the stack.

[mojodojodev]

For Julia, they've given quite detailed responses to that in the past here: https://mojodojo.dev/mojo-team-answers.html#julia

[hmcbride]

So to do large scale data prep and cleansing I will still have to use a tool like Spark to prepare data for a model , and then run it in Mojo is that correct ?

[mojodojodev]

Yeah that's right, Mojo is currently pretty focused addressing the limitations that make Python slow and error prone. It'll be a good candidate for large projects like distributed compute and machine learning frameworks in the future, but that'll likely come from community projects somewhere down the line.