📢Really happy to announce the "Scaling with Dask guide" for Panel. 🤝In many ways Panel and Dask is a perfect couple as they share both philosophies and key technologies. 👇Link to guide below #python #dataviz pandas Coiled #datascience #Analytics

New blog post on how to schedule Python jobs with Prefect + Coiled

Join us to see how to churn through many cloud files in parallel. We’ll cover: - Adapting existing code with minimal changes - Cost optimization - Auto-scaling to process TBs of data in parallel 🗓 Wednesday, January 31st ⏰ 11 am Eastern Register: content.coiled.io/parallel-cloud…

Join us next week to see how to schedule Python jobs with Prefect and Coiled. Prefect makes it easy to write production workflows in Python. Coiled makes it easy to deploy Prefect in the cloud. Register for the webinar: content.coiled.io/prefect-webinar

Query planning for Dask Dataframe released github.com/dask/dask/issu… Dask dataframe is pretty fast now, even when folks write less-than-perfect code. $ pip/conda/uv/mamba install dask

On the 14th of May, QuantCo Karlsruhe will host the next PyData Südwest Meetup. Florian Jetter will join us to talk about Dask's impressive speed, and Pavel, Adrian, and Bela show how to manage hundreds of Python Sign up at meetup.com/pydata-suedwes…

Recommendation of the day: `coiled notebook start` to run a remote Jupyter Lab from big machines in cloud but with file sync that feel "local". Demo from Coiled youtu.be/mibhDHYun0M #python #jupyter

TPC-H Cloud Benchmarks: Spark, Dask, DuckDB, Polars Across scales: 10 GiB, 100 GiB, 1 TiB, 10 TiB Hardware: MBP and AWS It was a fun experiment. No project wins uniformly. DuckDB and Dask do pretty well. docs.coiled.io/blog/tpch.html

Dask DataFrame is now 20x faster. Some of most prominent changes include: - Apache Arrow support in pandas - Better shuffling algorithm for faster joins - Automatic query optimization Learn more: docs.coiled.io/blog/dask-data…

Run a Python script on a cloud GPU with one line of code. Training a PyTorch model training takes ~10 minutes and cost ~$0.12 on the NVIDIA T4 GPU on AWS. Coiled handles provisioning hardware, setting up drivers, and installing CUDA-compiled PyTorch. docs.coiled.io/user_guide/gpu…

Arraylake and Coiled work great together! You can use Coiled to manage your cloud computing infrastructure with Dask, and store your data as zarr_dev in Arraylake. We just added new a documentation page about our integration with Coiled. docs.earthmover.io/integrations/c…

We're to build a 100-TB scale geospatial benchmark suite docs.coiled.io/blog/geospatia… We've seen an uptick in geospatial users and in challenges of the Xarray/Dask stack to scale beyond ~500-GiB. This post presents a call for benchmark workloads.

Implemented Coiled into our product to offload data syncing from BigQuery to Neo4j 🤯 Works like butter 🧈 Now I don’t have to worry about scaling VMs dynamically to handle variable loads.

New blog post: Scale AI-based Data Processing EASY The FineWeb-Edu dataset comes from processing 45TB (🤯) of FineWeb And it uses a Language Model to classify the educational level of the text 😭😭 Still, we reproduced it in a few lines of code ! The key ? HF + Dask 😎