Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
nerdponx
on Feb 10, 2023
|
parent
|
context
|
favorite
| on:
DuckDB – An in-process SQL OLAP database managemen...
Pandas doesn't. Polars I think has some lazy-loading capability, but it's not the default mode of operation and I don't think it supports all features. If DuckDB doesn't, then that's a big advantage.
jcheng
on Feb 11, 2023
|
next
[–]
I think you mean that Pandas
does
require your entire data to fit in memory?
https://pandas.pydata.org/docs/user_guide/scale.html
nerdponx
on Feb 11, 2023
|
parent
|
next
[–]
Yes, thank you, that was a typo.
corford
on Feb 11, 2023
|
prev
[–]
Apparently duckdb paired with arrow lets you work performatively on bigger than memory parquet data (I haven't tried this though)
riezebos
on Feb 11, 2023
|
parent
[–]
Arrow seems to be the common denominator for larger than ram. Polars is completely built on arrow, so the same advantages should apply.
https://github.com/pola-rs/polars#handles-larger-than-ram-da...
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
