Changelog

Unreleased

0.19.0 - 2026-09-03

Added

• KafkaReader.fast_forward: skip the assigned partitions to the first message at or after a GPS time (or to the end), dropping the backlog, for a consumer that has fallen further behind real time than it can catch up
• Live direct-Kafka streams skip ahead automatically when stuck behind their latency budget, trading the unusable backlog for a logged gap; Client.stream(fast_forward=False) opts out. Bounded and server-relayed streams are unaffected

Changed

• Publisher.publish now collects delivery reports and raises the new arrakis.PublishError when delivery fails (e.g. a leaderless partition), instead of silently dropping blocks. The previously unused timeout argument bounds delivery confirmation, raising TimeoutError; by default publishing blocks until every message resolves. A failed block may be retried
• The publisher flushes once per block instead of per partition, bounds its flush at close, and routes librdkafka's logs into the arrakis logger instead of stderr

0.18.2 - 2026-08-27

Fixed

• A Kafka reader spanning many partitions with fast channels overran its 4 MB prefetch cap every stride, and each overrun paused fetching for a second, so its data arrived at the edge of the latency contract and was partly dropped as late. The cap now scales with the number of partitions (4–64 MB) and an overrun pauses fetching for 100 ms

0.18.1 - 2026-08-25

Fixed

• BlockMuxStream.ready() cost grew with the square of the number of streams, leaving a live subscription over many partitions served seconds late; its per-stream pass now runs once per stride
• The muxer's stream-outage warning printed the gap's GPS time as a duration in nanoseconds, and fired for every gap run however short. It now reports a stream once its gap run spans the latency contract (at least 1 s), in seconds; shorter runs only leave a DEBUG line, since the late-drop summary already covers marginal lateness

0.18.0 - 2026-08-25

Added

• EndpointStalledError: a live Flight endpoint that goes silent past its latency contract now fails the stream instead of being gap-filled indefinitely, so a dead connection is no longer indistinguishable from a publisher dropout
• label on BlockMuxStream, prefixing its queues' names in log messages with the caller's context

• Cli

  --quiet/-q and --verbose/-v, overriding ARRAKIS_LOG_LEVEL

Changed

• Stream-health logging is transition-based and quiet at INFO: a stream that goes quiet after delivering data warns once and logs recovery once, transient Kafka consume errors are rate-limited per episode, and muxer log lines name streams by their channels
• Client.stream and Publisher log one INFO line on open; raw producer properties move to DEBUG
• The warn drop policy no longer logs every drop on top of the rate-limited summary

• Cli

  -v means --verbose (--version keeps only its long form), log lines include the level, and the stderr handler is attached once per process

Fixed

• MultiFlightReader.done() no longer reports a stream whose endpoints died with errors as a clean end of stream

0.17.1 - 2026-08-18

Changed

• Rate-limit the block muxer's stride-mismatch warning: the first dropped block warns immediately, sustained drops are summarized at most once per interval per stream (via TimedQueue.count_drop, now public), and recovery is logged with the unreported drop count when matching blocks resume, closing the episode so a later mismatch warns immediately again

Fixed

• Send HTTP/2 keepalive pings no faster than the ping policing of a default-configured gRPC server allows: the previous 30 s interval drew a "Too many pings" disconnect on streams with no data flowing, tearing down quiet-but-healthy connections

0.17.0 - 2026-08-11

Added

• Add Bearer token authentication for all Flight connections — automatic token discovery via igwn-auth-utils, explicit token strings, or an explicitly unauthenticated mode — with credentials relayed through the Flight SSH proxy
• Add KafkaReader.exhausted(), reporting when every assigned partition has been consumed to its high watermark, and BlockMuxStream.declare_complete(), finalizing spans without data as gaps — together giving bounded reads a terminal state over absent data instead of waiting forever for data that cannot arrive

Fixed

• Cancel in-flight reads when closing a stream: closing a stream with no data flowing now returns promptly instead of blocking until the next block arrives, and interrupting a quiet reader no longer hangs the process at exit

0.16.0 - 2026-08-05

Added

• Add a stride parameter to Client.stream, arrakis.stream and the stream CLI (--stride) for requesting output blocks covering an integer multiple of the native stride, aggregated client-side
• Add Channel.from_validated, a trusted fast-path constructor that skips validation for already-validated sources, speeding up bulk channel loads ~3x

Fixed

• Skip Kafka messages with no rows for the requested channels (e.g. a publisher running with a stale partition assignment) with a warning instead of crashing the reader
• Do not log expected live startup alignment trims as late drops
• Skip the publisher unit tests when pytest-arrakis is not installed instead of failing collection (fixes packaging test environments)

0.15.0 - 2026-07-27

Added

• Stamp the request schema version into every request envelope, so servers can report actionable errors for requests newer than they support instead of failing opaquely
• Add parse_request(), returning a ParsedRequest envelope with the request type, arguments, and the sender's schema version (None for requests predating version stamping), and RequestValidator.known_args(), exposing the schema-defined argument names per request type — the parsing surface servers use to reject unknown arguments explicitly

Fixed

• Make Channel.__hash__ consistent with its equality semantics: equal channels now hash equally even when optional fields (publisher, partition info, timing) are defined on only one side, so channels work correctly in sets and as dict keys
• Compare zero-valued partition_index, stride and max_latency in Channel equality; truthiness checks previously skipped them

Changed

• Require arrakis-schema >= 0.5

0.14.1 - 2026-07-22

Fixed

• Raise a failed endpoint stream's error on the next read instead of deferring it to stream close, where it was indistinguishable from a data gap while the muxer filled masked blocks
• Improve Flight SSH proxy robustness: preserve endpoint and flight-info metadata when rewriting endpoints, verify the proxy server started before yielding, time out the port-forward wait instead of blocking indefinitely, handle plain (non-URL) SSH destinations, and clean up the backend client and leftover control sockets on shutdown

Changed

• Apply gRPC HTTP/2 keepalive options (30s ping on idle transports, 10s ack timeout) to all Flight connections, so dead servers and dropped network paths surface as connection errors instead of indefinite hangs; callers can override via generic_options
• Improve the publisher registration error when no channels match: the message now states that no channels are registered for the publisher, including the replay context queried, instead of "unknown publisher ID"
• The Flight SSH proxy rejects publish RPCs (DoPut/DoExchange) with a clear error, since publishing through the proxy is intentionally unsupported

0.14.0 - 2026-07-14

Added

• Add on_gap policy ("fill", "skip", "raise") to stream() and fetch() for controlling how spans with no data are handled; clients default to filling gaps with masked values
• Add ARRAKIS_KAFKA_DEBUG environment variable to enable librdkafka debug logging in Kafka consumers

Fixed

• Fix live streams degrading to all-gap blocks when a single channel exceeds its latency budget — server-relayed streams now get timeout headroom in the client muxer so only the late channel is masked
• Fix duplicate and non-contiguous blocks at stream start — the muxer no longer rewinds its serving position when data arrives for slots already served
• Fix intermittent hang when resolving start offsets for bounded reads — offset lookups are retried under a deadline with fallbacks to the earliest available offset
• Fix mid-stream schema errors when multiple clients request the same channel set in different orders — stream schema field order is now canonical (sorted by channel name)
• Fix precision errors in replay time translation by using exact integer arithmetic

Changed

• The muxer no longer raises on queue overflow or time discontinuities — affected data is dropped and summarized in rate-limited log messages
• The Kafka reader only raises on errors librdkafka marks as fatal; transient conditions (missing topics, broker rebalances) are logged and retried, and missing topics are picked up automatically once they appear
• Change the default on_drop policy from "warn" to "ignore", with dropped data reported in rate-limited summaries
• Kafka consumers now assign partitions directly instead of subscribing through consumer groups, removing the coordinator dependency, rebalance churn, and throwaway consumer groups
• Reduce muxer memory and processing overhead during outages — gaps are synthesized on demand instead of queued as placeholder entries, with identical streaming output

0.13.1 - 2026-07-06

Fixed

• Fix client hang when a single-stream request times out — the muxer now gap-fills single-queue streams when the server exceeds max_latency instead of hanging indefinitely

Changed

• Reduce muxer memory footprint by scaling the per-queue safety cap to the timeout window instead of a fixed 1M-element limit
• Reduce Kafka consumer memory footprint by lowering librdkafka internal fetch buffer limits (queued.min.messages, queued.max.messages.kbytes)

0.13.0 - 2026-06-17

Added

• Add replays() method to Client and top-level API for querying available replay windows from the server
• Add HasStreams protocol and return type annotations for StreamReader implementations
• Support multi-broker kafka:// endpoint locations — KafkaReader now connects with the full broker list instead of only the first broker

• Cli

  add replays subcommand

• Cli

  add domains and scope-map subcommands

• Cli

  add --time flag to find, count, and describe for metadata-at-time queries

• Cli

  add --replay-id flag to publish for publishing under a replay context

• Cli

  add --duration option to stream as an alternative to --end

• Cli

  allow --data-type and --publisher to be specified multiple times

Fixed

• Cli

  fix formatting for null find result

• Cli

  fix IndexError when channel lists read from stdin contain blank lines

• Fix block table title formatting
• Fix default value handling when getting field justification

Changed

• Rework SSH proxy interface — proxy is now activated via --url ssh:// and the proxy subcommand has been renamed to exec
• Improve parse_arrakis_url: handle missing scheme, missing host/port defaults, and return a ParseResult object; moved to client module
• Improve SSH proxy bind_address parsing and ssh:// scheme handling

• Cli

  rename --min_rate/--max_rate to --min-rate/--max-rate for consistency

0.12.0 - 2026-06-01

Added

• Add replay support to stream() and fetch() via replay_start/replay_end or replay_id for streaming archival data retimestamped as live
• Add replay_id parameter to find(), count(), and describe() for querying channels published under a named replay context
• Add replay_id support to Publisher for producing derived channels under a replay namespace, with replay-specific Kafka topic routing
• Add arrakis.replay module with calculate_offset and translate_time utilities shared between client and server
• Add Client.endpoint() method to return server endpoints
• Add FlightProxy server for improved SSH proxy support

• Cli

  add server-info subcommand

• Cli

  add --replay-id flag to find, count, describe, stream, and fetch; add --replay-start/--replay-end flags to stream and fetch

Changed

• Cli

  improve table formatting output

• Cli

  catch KeyboardInterrupt instead of using signal handler

0.11.1 - 2026-05-04

Fixed

• Fix issue muxing data from streams with different strides, which caused data to be unintentionally dropped

0.11.0 - 2026-04-20

Added

• Publisher: support self-registration for "dynamic" publishers — pass channels to Publisher.register(channels=...) and the server will dynamically partition any new or changed channels
• Add scope_map() method to Client and the top-level API for querying endpoint scope information from the server
• Add MultiStreamReader for merging multiple underlying stream readers into a single stream, enabling stream requests that span multiple Kafka broker endpoints (e.g. per-publisher brokers)
• Series: add .array, .mask, and .has_gaps properties for a clearer masked-array API; also add .has_gaps on SeriesBlock

Fixed

• Publisher: fix latent bug in partition_channels where the Flight exchange was run against an already-closed connection

Changed

• Rename publish optional install extra to cli to better reflect where its dependencies are needed
• Overhaul user documentation: new getting-started tutorial, user guides for connecting/finding/fetching/streaming/publishing, and background pages covering architecture and the data model

Removed

• Series.has_nulls — use the new .has_gaps property instead

0.10.0 - 2026-04-02

Added

• Add server_info() and domains() methods to Client and top-level API for querying server metadata and available domains
• Add ChainedFlightReader for sequential reading across time-ordered endpoint slices, replacing MultiFlightReader for fetch requests
• Add time parameter to find(), count(), and describe() for time-aware metadata queries against historical backends

• Cli

  add proxy command for wrapping subprocesses in an SSH proxy

• Cli

  add format options for publish --list

Fixed

• BlockMuxStream: reject blocks with mismatched strides instead of silently muxing incorrect data
• TimedQueue: allow partial-stride edge blocks for non-aligned requests
• Channel: coerce all numeric-like dtype arguments to strings, not just numpy.dtype instances
• Support historical streams without max_latency — no longer required for channels when streaming

Changed

• Publisher: add validation checks for monotonically increasing timestamps and consistent block stride; improve error messages for metadata mismatches
• Bump minimum arrakis-schema version to 0.3

• Cli

  rename log level environment variable from LOG_LEVEL to ARRAKIS_LOG_LEVEL, default to INFO

0.9.0 - 2026-03-26

Added

• Cli

  add table output format using rich, with a live-updating table for arrakis stream output

• Cli

  add ability to read channels from stdin

• Cli

  add --format option to count command

Fixed

• TimedQueue: fix drain_until not advancing last_time, causing queue misalignment
• TimedQueue: clear stale timeout gaps on first real data
• TimedQueue: reset last_time when queue is drained instead of preserving old value
• Guard pull() with ready() when reader is done in BlockMuxStream.stream()
• Skip timeout gap-filling when there is only a single queue
• Update timeouts before aligning queues in muxer

Changed

• Cli

  make table the default output format for all commands

• TimedQueue: make update_timeout() public for use in muxer

0.8.0 - 2026-03-11

Added

• Cli

  add --duration option to fetch as an alternative to --end

• Cli

  add --ssh-proxy option to tunnel connections through an SSH host

Fixed

• Fix issue with duplicate gap channels in blocks in the muxer
• Fix muxer logic in BlockMuxStream.stream() to yield blocks as they become ready during reads, improving latency for streaming data

Changed

• Remove old Muxer and MuxedData classes, replaced by BlockMuxStream

0.7.0 - 2026-01-26

Added

• Add functionality to Channel class:
• Add as_dict method to Channel
• Add stride and max_latency attributes
• Add fields() method to return list of attribute fields
• Add SeriesBlock.full_gap to create a full-gap block
• Add environment variable for specifying ON DROP behavior in muxer
• Add start_ns and end_ns properties for SeriesBlock
• Add KafkaReader for reading data directly through Kafka
• Introduce new StreamReader protocol for asynchronously reading streams. This can seamlessly read data from Arrow Flight RPC or Kafka through the stream endpoint

Fixed

• Fix nanosecond conversion in time_as_ns
• SeriesBlock: fix mask when creating a pyarrow array from a numpy masked array
• Fix block concatenation for masked arrays
• Fix publish command in CLI to properly use the new publisher interface

Changed

• Allow channels to have names with only subsystems
• Forbid None as a valid data type. This used to be previously interpreted as a float64 based on how numpy can interpret None
• Move Flight endpoint schemas from the server to the client
• Update publisher interface:
• Register step is now used solely to resolve channels to be published
• Enter method now handles retrieving all Kafka producer information
• Remove PublisherInfo class as the publisher now has all the partition info after the registration step
• Improve Channel repr: better formatting for int parameters, include more information
• Specify default output format of 'str' for CLI
• Improve logging of get_flight_info
• Allow gaps to be filled in data streams after reaching the timeout to avoid hanging clients through the new muxer interface, BlockMuxStream

0.6.1 - 2025-11-06

Fixed

• Fix issues with publication with partition index scheme
• Extract partition ID as well for relevant publishing info
• Fix validation check in partition index for registration
• Extract partition index from metadata in Client, fixes issue where find/describe had missing partition index
• Raise correct RuntimeError instead of unclear AttributeError when publishing without context manager

0.6.0 - 2025-10-28

Added

• Add partition_index attribute to Channel class

Changed

• Change allowable channel name structure: <domain>:<subsystem>[-_]<rest>
• This allows VIRGO-like channels to be parsed correctly
• Also expose subsystem property to Channel class
• Update publisher to push channel partition index values instead of names, reducing packet sizes
• Track channel name to ID values during registration and partitioning
• SeriesBlock.from_row_batch takes a partition index - channel map instead

0.5.0 - 2025-09-26

Added

• Allow a pre-defined schema to be passed into SeriesBlock.to_column_batch

Fixed

• Address muxer edge cases causing stale data to not be returned
• Fix edge case in muxer when we get complete data for a newer timestamp after incomplete data from an older timestamp

Changed

• Improve performance of SeriesBlock generation from record batches:
• Make fath path quicker for non-null Arrow arrays in converting to numpy arrays
• Avoid unnecessary Arrow array type inference
• Extract single time from batch instead of converting to numpy array first
• Switch to more efficient bit manipulation to calculate Arrow array mask for numpy conversion
• Improve performance of conversion from numpy masked arrays to Arrow arrays for nested types

0.4.1 - 2025-07-10

Fixed

• Fix edge case in muxer where multiple blocks with the same time could be returned

0.4.0 - 2025-06-25

Added

• Add support for gaps in data, represented as masked arrays in SeriesBlock
• Add property in Series that reports gaps in data
• Allow printing channel as JSON from CLI in arrakis describe/find
• Add --latency option to print buffer latency to stderr
• Add 'expected latency' metadata in Channel
• Allow creation of gaps within SeriesBlock to support server-side gap handling

Fixed

• Fix match parsing on drop in muxer
• Fix issue where items in muxer when setting on_drop to 'warn' was not dropping items

0.3.0 - 2025-04-16

Added

• Add option to specify URL in arrakis CLI
• Add dtype alias to data_type in Channel
• Add publish sub-command in Arrakis CLI to generate arbitrary streams to publish to the specified channels
• Add schema validation to request descriptors for client and server-side validation

Fixed

• Add min_rate/max_rate arguments if not specified in client, addressing a failure if specified as None
• Fix issue in excessive CPU usage when polling MultiEndpointStream
• Coerce data types to strings within Client so they are JSON-serializable
• Fix describe command in arrakis CLI to properly extract channel info for display

Changed

• Use GPSTimeParseAction for time arguments/options in arrakis CLI, allowing arbitrary date/time strings
• Redefine eq for Channel, relaxing strict equality for optional fields
• Update publication interface:
• take the publisher_id at initialization, not during register
• the register step now retrieves the channel list and updates the partition info
• context manager now handles retrieving kafka info from the server to allow publication
• publish method checks consistency of channels being published
• Check channels when initializing publisher

0.2.0 - 2025-03-11

Added

• Add publisher metadata to Channel
• Allow multiple data types in find/count requests
• Allow querying by publisher in find/count requests
• Add from_json constructor in Channel
• Add arrakis entry point

Fixed

• Fix issue in parsing response in Publisher registration
• Improve error handling and mitigate timeouts in MultiEndpointStream polling
• Remove initial describe call within stream endpoint

Changed

• Allow Channel to handle raw numpy dtypes
• Expose domain property for Channel
• Publisher now only requires a publisher_id for registration

Removed

0.1.0 - 2024-11-13

• Initial release.