Production
Work that runs without you watching it.
Topics in Production
5 topics, 140 guides. Each topic page groups the concepts, the how-to guides and the fixes for that subject.
-
Batch processing
Running one operation across a whole folder without babysitting it.
4 concepts · 25 how-to · 7 fixes → -
Pipelines & automation
Config-driven, scheduled, observable workflows that run without you.
12 concepts · 20 how-to · 8 fixes → -
Performance & scale
Spatial indexes, parallelism, memory, and datasets too big to load at once.
9 concepts · 14 how-to · 11 fixes → -
Cloud-native formats
COG, Zarr, GeoParquet and STAC: reading the part you need over the network instead of downloading the file.
4 concepts · 8 how-to · 3 fixes → -
Testing, CI & Docker
Proving a GIS workflow still works, and shipping it somewhere reproducible.
4 concepts · 6 how-to · 5 fixes →
Every guide in Production
Grouped by topic. A guide that spans two topics appears under both.
Batch processing 36 guides
- How to Batch Process Multiple Shapefiles in Python batch process shapefiles python
- How to Batch Process a Folder of GIS Files in Python: The Complete Workflow batch process gis files python
- How to Batch Convert Shapefiles to GeoPackage in Python batch convert shapefile to geopackage python
- How to Batch Clip Many Layers to a Single Boundary in Python batch clip shapefiles boundary python geopandas
- How to Merge Many Shapefiles into One File in Python merge multiple shapefiles python geopandas
- How to Batch Export a GeoDataFrame to Multiple Formats in Python export geodataframe multiple formats python
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python parallel processing gis python batch
- How to Build a Resumable Batch GIS Job in Python resumable batch job python gis
- How to Log and Summarise Errors in a Batch GIS Job in Python python logging batch gis errors report
- How to Batch Process Rasters with Rasterio in Python batch process rasters rasterio python
- How to Process GIS Files in Nested Subfolders and Mirror the Output Tree process files subfolders python recursive mirror
- How to Batch Process Layers in QGIS with PyQGIS batch process layers pyqgis
- Batch Script Stops at the First Bad File: How to Keep It Running batch script stops on error
- Python glob Is Not Finding All My Shapefiles: How to Fix It glob not finding files python
- Batch Output Files Keep Overwriting Each Other: How to Fix It batch output files overwriting
- Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It batch job memory leak python
- Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It multiprocessing hangs python gis
- How to Add a Progress Bar and ETA to a Long GIS Batch Job python progress bar batch job
- How to Build an Inventory of a GIS Data Folder in Python gis data folder inventory python
- How to Add a Dry-Run Mode to a Batch GIS Script dry run mode python script
- How to Package GIS Deliverables into Zipped Bundles with Python package gis deliverables zip python
- How to Process a Very Large GeoPackage in Chunks with Python process large geopackage in chunks
- The Anatomy of a Batch Job: Discover, Apply, Report batch job structure python
- Failure Policy in Batch Processing: Fail Fast, Skip or Quarantine batch error handling policy
- Choosing the Unit of Work in a Batch Job: File, Layer, Feature or Tile batch unit of work gis
- Incremental Processing: How a Batch Job Knows What Changed incremental batch processing gis
- How to Process Only the Files That Changed Since the Last Run process only changed files python
- How to Split a Large Layer into Tiles for Batch Processing split layer into tiles python
- How to Enrich Many Files Against One Reference Layer batch spatial join reference layer
- Batch Job Gives Different Results Each Run: How to Fix It gis batch non deterministic output
- OSError: Too Many Open Files in a Batch GIS Job too many open files python gis
- How to Merge and Mosaic Rasters in Python with Rasterio merge rasters rasterio mosaic
- How to Load a Folder of Shapefiles into PostGIS with Python load shapefiles into postgis python
- How to Batch-Generate a Map Image for Every Region in Python batch generate maps python
- How to Clip and Tile a Point Cloud in Python clip tile point cloud python
- How to Export a Static Tile Pyramid for Offline Use export static tile pyramid python
Pipelines & automation 40 guides
- How to Automate GIS Workflows with Python automate gis workflow python
- How to Build a GIS Data Pipeline in Python: The Complete Workflow gis data pipeline python
- How to Chain GIS Processing Steps into a Reusable Pipeline in Python chain gis processing steps python
- How to Drive a GIS Pipeline from a YAML Config File in Python yaml config gis pipeline python
- How to Turn a GIS Script into a Command-Line Tool with argparse python gis script command line argparse
- How to Schedule a Python GIS Script to Run Automatically schedule python gis script cron
- How to Add Retries and Timeouts to an Automated GIS Job in Python python retry timeout gis job
- How to Validate Pipeline Inputs and Outputs Automatically in Python validate gis pipeline inputs outputs
- How to Cache Pipeline Steps So Only Changed Data Is Reprocessed cache gis pipeline steps python
- How to Get Alerted When an Automated GIS Job Fails alert gis job failure python
- How to Make a GIS Workflow Reproducible in Python reproducible gis workflow python
- Python GIS Script Works Manually but Not from Cron: How to Fix It cron python script not running
- ModuleNotFoundError in a Scheduled GIS Job: Fixing the Wrong Python Environment modulenotfounderror scheduled script
- Relative Paths Break When a GIS Script Runs Automatically: How to Fix It relative path fails in cron python
- YAML Config File Will Not Load: Common Errors in a Python GIS Pipeline yaml config error python
- My GIS Pipeline Fails Silently: Fixing Swallowed Errors and Wrong Exit Codes python script fails silently
- How to Containerise a Python GIS Pipeline with Docker docker python gis pipeline
- How to Test a GIS Pipeline with pytest test gis pipeline pytest
- How to Record Run Metadata and Data Lineage in a GIS Pipeline data lineage python gis pipeline
- How to Handle Credentials and Secrets in an Automated GIS Job python secrets credentials gis job
- How to Run a Python GIS Pipeline in CI with GitHub Actions github actions python gis pipeline
- What Is a GIS Data Pipeline? The Anatomy Explained what is a gis data pipeline
- Idempotency Explained: Why a GIS Job Must Be Safe to Re-run idempotency data pipeline explained
- Configuration vs Code: What Belongs in a Config File configuration vs code
- Logs, Metrics and Alerts: Observability for GIS Pipelines observability data pipeline logs metrics
- Cron, systemd, Airflow or CI? Choosing a Scheduler for GIS Jobs choosing a scheduler for python jobs
- What to Test in a GIS Pipeline (and What You Cannot) what to test gis pipeline
- GIS Tests Pass Locally but Fail in CI: How to Fix It gis tests fail in ci
- The Anatomy of a Batch Job: Discover, Apply, Report batch job structure python
- Failure Policy in Batch Processing: Fail Fast, Skip or Quarantine batch error handling policy
- pytest Cannot Import My GIS Modules: How to Fix It pytest modulenotfounderror src
- GeoPandas Cannot Connect to PostGIS: How to Fix It geopandas postgis connection error
- Repair, Reject or Flag? Choosing What Cleaning Should Do gis data cleaning decisions
- Choosing the Unit of Work in a Batch Job: File, Layer, Feature or Tile batch unit of work gis
- Incremental Processing: How a Batch Job Knows What Changed incremental batch processing gis
- How to Process Only the Files That Changed Since the Last Run process only changed files python
- How to Update a PostGIS Table from a GeoDataFrame Without Duplicates postgis upsert geodataframe
- How to Profile a Slow Python GIS Script and Find the Real Bottleneck profile slow python gis script
- How to Cache Downloaded GIS Data So You Fetch It Once cache downloaded gis data
- How to Publish a GeoJSON API from a GeoDataFrame with FastAPI geojson api fastapi python
Performance & scale 34 guides
- How to Speed Up GeoPandas: Tips for Large Datasets geopandas performance optimization
- Fixing Memory Errors in GeoPandas When Working with Large Files geopandas memory error
- How to Speed Up Batch GIS Jobs with Parallel Processing in Python parallel processing gis python batch
- How to Cache Pipeline Steps So Only Changed Data Is Reprocessed cache gis pipeline steps python
- Batch GIS Job Slows Down and Runs Out of Memory: How to Fix It batch job memory leak python
- Parallel GIS Batch Job Hangs or Crashes with multiprocessing: How to Fix It multiprocessing hangs python gis
- How to Reduce GIS File Size in Python Without Wrecking the Data reduce gis file size python
- How to Process a Very Large GeoPackage in Chunks with Python process large geopackage in chunks
- Spatial Indexes Explained: R-trees and Why Spatial Joins Are Fast spatial index r-tree explained
- Threads, Processes and the GIL: Parallelism in Python GIS Explained python gil threads processes gis
- How to Split a Large Layer into Tiles for Batch Processing split layer into tiles python
- OSError: Too Many Open Files in a Batch GIS Job too many open files python gis
- Spatial SQL or GeoPandas? Choosing Where the Work Happens spatial sql vs geopandas
- PostGIS Spatial Indexes Explained: GiST and the Two-Phase Filter postgis gist spatial index
- How to Read a Large PostGIS Table into Python Without Running Out of Memory read large postgis table python
- My PostGIS Spatial Query Is Slow: How to Fix It postgis spatial query slow
- How to Build a Small, Fast Python GIS Docker Image small python gis docker image
- GIS Tests Run Out of Memory or Time Out in CI: How to Fix It ci out of memory gis tests
- Why GeoPandas Is Slow: The Four Real Bottlenecks why is geopandas slow
- GeoParquet and Columnar Storage Explained: Why It Reads So Much Faster geoparquet explained python
- How to Replace a Row Loop with Vectorised GeoPandas Operations vectorise geopandas loop
- How to Use the Spatial Index Directly in GeoPandas (sindex) geopandas sindex spatial index
- How to Profile a Slow Python GIS Script and Find the Real Bottleneck profile slow python gis script
- GeoPandas .apply() Takes Hours: How to Fix It geopandas apply slow
- Nearest-Neighbour Joins Explained: Distance, Ties and Search Radius nearest neighbour join explained
- How to Build a Distance Matrix Between Two Layers in Python distance matrix geopandas
- How to Measure Network Distance for Many Origin-Destination Pairs network distance matrix python
- Reading a COG from a URL Is Slow or Downloads Everything cog read is slow
- Lazy Loading Explained: Why xarray Reads Nothing Until You Ask lazy loading xarray explained
- Chunked Arrays and Zarr Explained zarr chunked arrays explained
- How to Choose Chunk and Tile Sizes That Actually Help choose chunk and tile sizes
- How to Scale a GeoPandas Job with dask-geopandas dask geopandas scale python
- dask-geopandas Uses More Memory or Time Than Plain GeoPandas dask geopandas slower than geopandas
- My GeoJSON Is Too Big for the Browser geojson too big for browser
Cloud-native formats 15 guides
- Cloud-Native Geospatial Explained: Why the Format Became the API cloud native geospatial explained
- Cloud-Optimised GeoTIFF Explained: Tiles, Overviews and Range Requests cloud optimised geotiff explained
- How to Read a COG from a URL Without Downloading the Whole File read cog from url python
- Reading a COG from a URL Is Slow or Downloads Everything cog read is slow
- How to Write a Cloud-Optimised GeoTIFF in Python write cloud optimised geotiff python
- How to Read Spatial Data from S3 and Other Object Storage read spatial data from s3 python
- GDAL Cannot Open an S3 or HTTPS Path gdal cannot open s3 url
- Lazy Loading Explained: Why xarray Reads Nothing Until You Ask lazy loading xarray explained
- Chunked Arrays and Zarr Explained zarr chunked arrays explained
- How to Turn a STAC Search into an xarray Data Cube stac to xarray cube python
- How to Choose Chunk and Tile Sizes That Actually Help choose chunk and tile sizes
- How to Write Partitioned GeoParquet and Query It with Filters partitioned geoparquet python
- How to Scale a GeoPandas Job with dask-geopandas dask geopandas scale python
- dask-geopandas Uses More Memory or Time Than Plain GeoPandas dask geopandas slower than geopandas
- How to Render Raster Tiles from a COG in Python raster tiles from cog
Testing, CI & Docker 15 guides
- How to Make a GIS Workflow Reproducible in Python reproducible gis workflow python
- How to Containerise a Python GIS Pipeline with Docker docker python gis pipeline
- How to Test a GIS Pipeline with pytest test gis pipeline pytest
- How to Run a Python GIS Pipeline in CI with GitHub Actions github actions python gis pipeline
- What to Test in a GIS Pipeline (and What You Cannot) what to test gis pipeline
- GIS Tests Pass Locally but Fail in CI: How to Fix It gis tests fail in ci
- pytest Cannot Import My GIS Modules: How to Fix It pytest modulenotfounderror src
- Batch Job Gives Different Results Each Run: How to Fix It gis batch non deterministic output
- Test Fixtures for GIS Code: Building Spatial Test Data You Can Trust gis test fixtures python
- Reproducible GIS Environments Explained: conda, pip, Lockfiles and Docker reproducible gis environment python
- How to Assert on Geometry in pytest Without Flaky Comparisons pytest assert geometry equal
- How to Test a GIS Script That Reads and Writes Files test gis file io pytest
- How to Build a Small, Fast Python GIS Docker Image small python gis docker image
- GDAL and GeoPandas Fail Inside Docker: How to Fix the Build geopandas gdal docker build fails
- GIS Tests Run Out of Memory or Time Out in CI: How to Fix It ci out of memory gis tests