What PeerDB-io/peerdb shipped
Written by FoxPlug from public releases; not affiliated with Peerdb. An automatic summary of the public release, pull request and commit data of github.com/PeerDB-io/peerdb. Peerdb did not write it and does not use or endorse FoxPlug. Every line links to the public change it describes.
Get a weekly update like this for your product, free
Week of September 21, 2026
What shipped
- Added structured ingestion processing for MongoDB with QRepValueIterator for BSON documents and SchemaProjector integration for CDC and QRep workflows. Pull request #4845
- Introduced
structured_ingestionanddrop_unexpected_valuesconfiguration options for MongoDB table mappings with validation throughout the mirror configuration pipeline. Pull request #4818 - BigQuery query CDC settings can now be updated live when resuming paused mirrors through optional
query_cdcpayload. Pull request #4846 - BigQuery CDC now uses maximum watermark column value as checkpoint and scans subsequent non-overlapping windows when bounded query windows are empty. Pull request #4829
- BigQuery query CDC poll windows are now configurable per mirror with
query_cdc_safety_lag_secondsandquery_cdc_max_query_window_secondssettings. Pull request #4819 - BigQuery CDC Avro staging files are now chunked to handle extremely large objects from long sync intervals. Pull request #4801
- Database migrations now run in a separate flow-migrate container on startup rather than automatically within Nexus. Pull request #4826
- Flow CI was reorganized into granular jobs with build tag support and strict unit test selection at package level. Pull request #4849
- MongoDB changestream errors are now retried to handle optimized execution path toggles in MongoDB 9.0. Pull request #4841
- BigQuery storage read API now works with appends and changes methods by projecting change stream columns with _PEERDB prefix. Pull request #4824
Why it matters
MongoDB structured ingestion is now fully integrated with CDC and QRep workflows, enabling more sophisticated data pipeline configurations. BigQuery CDC improvements include configurable poll windows, better checkpoint handling, and chunked file staging for large-scale syncs. Infrastructure changes to database migrations improve reliability and modularity.
Changelog entry
- feat(structured-ingestion-mongodb): DBI-1096 structured ingestion processing for MongoDB with QRepValueIterator and SchemaProjector integration Pull request #4845
- feat(structured-ingestion-mongodb): DBI-1096 structured ingestion settings through table mappings with validation Pull request #4818
- feat(bigquery): support live updates to query CDC settings when resuming paused mirrors Pull request #4846
- feat(bigquery): use maximum watermark column value as checkpoint with subsequent window scanning Pull request #4829
- feat(bigquery): make query CDC poll windows configurable per mirror Pull request #4819
- feat(bigquery): chunk BigQuery CDC Avro staging files for large sync intervals Pull request #4801
- feat(database): migrate to goose and run migrations in separate flow-migrate container Pull request #4826
- chore(ci): reorganize flow CI into granular jobs with build tag support and strict unit test selection Pull request #4849
- fix(mongodb): retry changestream errors for optimized execution path toggles Pull request #4841
- fix(bigquery): enable storage read API for appends and changes methods by projecting change stream columns Pull request #4824
- fix(clickhouse): skip table capacity check when getServerSetting access is denied Pull request #4838
- test(clickhouse): add explicit test cases for missing permissions validation Pull request #4842
- test(bigquery): add more e2e test scenarios for different ingestion methods Pull request #4831
MongoDB structured ingestion is live. BigQuery CDC gets configurable poll windows and smarter checkpoint handling. Database migrations now run in their own container for better reliability.
Three major areas shipped this week: MongoDB structured ingestion is now fully integrated with CDC and QRep workflows for more sophisticated configurations. BigQuery CDC received several improvements including live-updatable query settings, configurable per-mirror poll windows, and intelligent checkpoint management using maximum watermark values. Infrastructure upgrades moved database migrations to a separate container for improved reliability and modularity.
Week of September 14, 2026
What shipped
- BigQuery connector now uses storage read API client, improving throughput from 1.8 MB/s to ~18 MB/s for sources with heavy data volumes. Pull request #4797
- BigQuery CDC scheduling now uses
last_synced_atto determine when a sync completed successfully, improving reliability of scheduled pulls. Pull request #4810 - BigQuery connector now selects only configured columns via table mappings instead of SELECT * EXCEPT, reducing unnecessary data transfer. Pull request #4740
- Normalization now properly handles Nullable(JSON) columns, required for structured ingestion workflows. Pull request #4813
- MongoDB BSON to JSON conversion moved to flow/pkg, enabling ClickPipes' Discovery MongoDB schema inference. Pull request #4825
- Structured ingestion now accepts NaN and Inf float values, recording them as strings in intermediate raw events. Pull request #4798
- PostgreSQL CDC no longer performs unnecessary JSON marshal roundtrips for JSON columns, reducing CDC latency for large documents. Pull request #4759
- DropFlowSource cleanup retry alerts downgraded from errors to warnings to prevent alert spam during stuck cleanups. Pull request #4807
- MongoDB BSON to QValue converters refactored behind BsonToQValueConverter interface for better type dispatch. Pull request #4802
- ClickHouse allowed domains configuration now trims whitespace and ignores empty entries in comma-separated lists. Pull request #4795
Why it matters
This week includes significant performance improvements for BigQuery and PostgreSQL connectors that reduce latency and increase throughput for high-volume syncs. Structured ingestion support expanded with better JSON handling and MongoDB schema inference. Several reliability fixes address edge cases in CDC scheduling and error reporting.
Changelog entry
- BigQuery: Storage read API client now enabled by default, improving throughput approximately 10x for large datasets Pull request #4797
- BigQuery: CDC scheduling now uses
last_synced_atto detect successful syncs instead of justlast_attempt_atPull request #4810 - BigQuery: Table pulls now select only configured columns via mappings instead of SELECT * EXCEPT Pull request #4740
- PostgreSQL: CDC JSON column handling no longer performs unnecessary marshal/unmarshal roundtrips Pull request #4759
- Normalization: Nullable(JSON) columns now properly handled in normalize queries Pull request #4813
- Structured ingestion: NaN and Inf float values now accepted and recorded as strings Pull request #4798
- MongoDB: BSON to JSON conversion moved to flow/pkg for ClickPipes Discovery schema inference Pull request #4825
- MongoDB: BSON to QValue conversion refactored with per-type converters behind BsonToQValueConverter Pull request #4802
- Alerts: DropFlowSource cleanup retry failures downgraded from error to warning level Pull request #4807
- ClickHouse: Allowed domains configuration now trims whitespace and ignores empty entries Pull request #4795
- ClickHouse: Type combinations with Nullable and LowCardinality no longer require explicit listing Pull request #4799
- MongoDB: CDC now reports receive/process/addRecord timing metrics Pull request #4788
This week: BigQuery storage read API now live (18MB/s throughput), CDC scheduling reliability improved with last_synced_at tracking, PostgreSQL JSON handling optimized, and structured ingestion gains Nullable(JSON) support.
This week's improvements focus on performance and reliability. BigQuery connector throughput increased 10x with storage read API integration. PostgreSQL CDC latency reduced by eliminating unnecessary JSON parsing. BigQuery CDC scheduling now tracks successful syncs separately from attempts. Structured ingestion gains support for Nullable(JSON) columns and NaN/Inf float values. MongoDB schema inference enabled via BSON conversion refactoring.