Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> Instead of loading data into Pandas and doing a group by

SQL can do `group by`. No need for overkill with "data science tools".



If you have a csv, pandas is the "less overkill" route than loading it into a database to use sql.


Each their own tool, but that's nothing awk can't do [1]. I prefer a shell script, because shell is everywhere.

[1] https://stackoverflow.com/a/75073649


Sorry, but no, awk and shell scripts are not a good choice for this kind of work. I'm sorry, but they just aren't!

Sure, go for it if you're just doing stuff for your own personal interest. But if you're doing serious data-driven work in a professional environment, this is going to be an awful choice.


While definitely doable, using a SQL database to do data discovery is obtuse.


It's obtuse, but it's effective for some people. I know someone who hires devs from the .NET and Python space for finance. When asked to suggest a solution to any kind of data problem like this the interviewees split down the middle with the .NET ones almost entirely using a relational database and the Python ones using either the library du jour or suggesting something on the command line (since they often have Linux experience.)

In both parts of their org, the .NET area tends to use SQL exclusively for analysis and the Python folks use Pandas and bunch of other stuff. These departments are also in significantly different parts of their organization with their own mandates and culture.


As someone with +20 years in finance (hedge funds/trading) and knows .Net, Java, Python, C++, shell, etc, the first question I'd ask is: where's the data?

If I'm being asked to do data analysis, it's because we need the answer yesterday. So, the tool I choose always going to be a matter of which gets me the answer fastest and with the least amount of friction. That's almost always dictated by where the data is _now_. Not where it'd ideally be.


It's in this email attachment right here. Now what?


To me that sounds like a csv file and it'll likely be a python script from me.


Depends, that is what OLAP is all about, naturally most of the good tools happen to be commercial for the enterprise space.


Not really. DuckDB can operate on raw csv without ingesting them. This is as easy if not easier than setting up python and pandas.


Totally agree that it depends on the tools available but given a typical toolset (sql and python). I would lean towards python because some types of analysis are easier to express in code, especially when working on top of the multiple data sets.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: