Skip to content

sonic: enforce full config_db.json ownership#2297

Open
ideaship wants to merge 2 commits into
mainfrom
doc-sonic-generator-ownership-model
Open

sonic: enforce full config_db.json ownership#2297
ideaship wants to merge 2 commits into
mainfrom
doc-sonic-generator-ownership-model

Conversation

@ideaship
Copy link
Copy Markdown
Contributor

@ideaship ideaship commented May 20, 2026

Motivation

Regenerating a SONiC config from NetBox repeatedly raised the same question: when generate_sonic_config() rewrites config_db.json, which sections does it own, and may an operator hand-edit anything directly? PRs #2279 and #2290 both ran into it. The policy is now settled: the generator owns and generates config_db.json in full, and operators make no manual adjustments to it.

What this PR does

Documents that ownership model in the generate_sonic_config() docstring and brings the code in line with it.

  • Full ownership, documented. The docstring states whole-file ownership: every section the generator manages is rebuilt from NetBox data and hardcoded SONiC policy on each regen; no sections pass operator edits through unchanged. Two fields the generator does not itself emit — the DEVICE_METADATA localhost device attributes (e.g. device type) and the DATABASE schema VERSION — are inherited from the device's image-provided base config, not from operator hand-edits.
  • Owned tables rebuilt from scratch. Previously the generator deep-copied the base config and most helpers assigned only the keys they generate this run, so an entry removed from NetBox but still present in config_db.json (e.g. an old VLAN_MEMBER, BGP_GLOBALS VRF, VXLAN_TUNNEL_MAP, SNMP user, or syslog host) survived as stale, active config. The owned tables (OWNED_TABLE_KEYS — every generated table except the inherited DEVICE_METADATA and VERSIONS) are now cleared up front, so only regenerated entries remain.
  • BGP_GLOBALS['default'] replaced wholesale. It was written with key-level updates that merged into a pre-existing entry, so operator fields survived regen. It is now replaced wholesale, like every other generated VRF.
  • STATIC_ROUTE no longer preserves hand-edited routes. Clearing the owned tables drops pre-existing routes (e.g. an operator's custom blackhole or VRF route) before the management default route is written.

Tests

  • Illustrative helper tests (test_config_generator_ownership.py): _add_vrf_configuration and _add_vlan_configuration replace BGP_GLOBALS[vrf], VLAN entries, and ROUTE_REDISTRIBUTE keys wholesale.
  • Orchestrator tests (test_config_generator_orchestrator.py): pre-existing BGP_GLOBALS['default'] fields, STATIC_ROUTE entries, and stale owned-table entries are dropped on regen, while DEVICE_METADATA and VERSIONS survive.

@ideaship ideaship force-pushed the doc-sonic-generator-ownership-model branch from cae57c6 to 9952d93 Compare May 20, 2026 06:44
@ideaship ideaship requested a review from berendt May 20, 2026 06:45
@ideaship ideaship moved this from Ready to In review in Human Board May 20, 2026
@berendt berendt moved this from In review to In progress in Human Board May 26, 2026
@ideaship ideaship force-pushed the doc-sonic-generator-ownership-model branch from 9952d93 to f4c34e0 Compare May 28, 2026 08:25
@ideaship ideaship changed the title sonic: document and test config generator ownership model sonic: enforce full config_db.json ownership May 28, 2026
@ideaship ideaship moved this from In progress to In review in Human Board May 28, 2026
@ideaship ideaship marked this pull request as ready for review May 29, 2026 06:41
Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The ownership model is currently encoded via INHERITED_TABLE_KEYS + OWNED_TABLE_KEYS and duplicated table-name literals; consider centralizing these names (e.g., a single authoritative enum/set) and/or adding a small invariant check so new scaffold keys or ownership changes can’t silently drift from the documented model.
  • Given that OWNED_TABLE_KEYS is used to pop sections before scaffolding, it may be worth explicitly asserting in code (or a small helper) that every TOP_LEVEL_SCAFFOLD_KEY other than INHERITED_TABLE_KEYS is indeed in OWNED_TABLE_KEYS, so future additions don’t accidentally become neither owned nor inherited.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The ownership model is currently encoded via INHERITED_TABLE_KEYS + OWNED_TABLE_KEYS and duplicated table-name literals; consider centralizing these names (e.g., a single authoritative enum/set) and/or adding a small invariant check so new scaffold keys or ownership changes can’t silently drift from the documented model.
- Given that OWNED_TABLE_KEYS is used to pop sections before scaffolding, it may be worth explicitly asserting in code (or a small helper) that every TOP_LEVEL_SCAFFOLD_KEY other than INHERITED_TABLE_KEYS is indeed in OWNED_TABLE_KEYS, so future additions don’t accidentally become neither owned nor inherited.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@ideaship ideaship marked this pull request as draft May 29, 2026 08:38
@ideaship ideaship force-pushed the doc-sonic-generator-ownership-model branch from f4c34e0 to ba31c99 Compare May 29, 2026 08:51
@ideaship
Copy link
Copy Markdown
Contributor Author

Updated with constant split + taxonomy invariant tests. No functional change.

@ideaship ideaship marked this pull request as ready for review May 29, 2026 08:51
Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The ownership taxonomy (INHERITED_TABLE_KEYS, SCAFFOLDED_OWNED_TABLE_KEYS, ON_DEMAND_OWNED_TABLE_KEYS) is currently maintained by hand; consider centralizing the table names in a single source (e.g., a mapping from table → classification) so that adding/removing tables in helpers can’t silently fall out of sync with these constants and the docstring.
  • The generate_sonic_config docstring hard-codes several table names (BGP_GLOBALS, VLAN, VLAN_INTERFACE, etc.) that are also captured in OWNED_TABLE_KEYS; to avoid drift, it may be cleaner to either reference OWNED_TABLE_KEYS symbolically in the docstring or reduce the explicit list there and rely on the constants for the authoritative set.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The ownership taxonomy (INHERITED_TABLE_KEYS, SCAFFOLDED_OWNED_TABLE_KEYS, ON_DEMAND_OWNED_TABLE_KEYS) is currently maintained by hand; consider centralizing the table names in a single source (e.g., a mapping from table → classification) so that adding/removing tables in helpers can’t silently fall out of sync with these constants and the docstring.
- The generate_sonic_config docstring hard-codes several table names (BGP_GLOBALS, VLAN, VLAN_INTERFACE, etc.) that are also captured in OWNED_TABLE_KEYS; to avoid drift, it may be cleaner to either reference OWNED_TABLE_KEYS symbolically in the docstring or reduce the explicit list there and rely on the constants for the authoritative set.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

The SONiC config generator owns config_db.json in full: it generates the
file from NetBox data and hardcoded SONiC policy, and operators must not
make manual adjustments to it.  This ownership model was implicit and
undocumented, so each PR touching the generator relitigated which
sections it owns (surfaced in review of #2279 and #2290).

Document the model in the generate_sonic_config() docstring as whole-file
ownership: the generator overwrites every section it manages, and no
sections pass operator edits through unchanged.  Note the two fields the
generator does not itself emit and instead inherits from the device's
image-provided base config (the DEVICE_METADATA localhost device
attributes and the DATABASE schema VERSION); these come from the image,
not from operator hand-edits.

Bring the code into line with the documented model:

  - Rebuild owned tables from scratch instead of overwriting only the
    entries touched in the current run.  generate_sonic_config() deep-
    copies the base config and most helpers assign only the keys they
    currently generate, so an entry removed from NetBox but still present
    in config_db.json (e.g. an old VLAN_MEMBER, BGP_GLOBALS VRF, or
    VXLAN_TUNNEL_MAP) survived regen as stale, active config.  Drop the
    content of every owned table up front and only regenerated entries
    remain.  OWNED_TABLE_KEYS is composed from two named sub-categories:
    the scaffolded-owned tables (every scaffold key except the inherited
    DEVICE_METADATA and VERSIONS, created up front via setdefault) and
    the on-demand owned tables (ROUTE_REDISTRIBUTE, the SNMP_* tables and
    SYSLOG_SERVER, which helpers assign only when NetBox carries their
    data).
  - BGP_GLOBALS['default'] was written with key-level updates that merged
    into any pre-existing entry, so operator fields survived regen.
    Replace the entry wholesale when a primary IP is present, like every
    other generated VRF.
  - STATIC_ROUTE merged the management default route into the dict loaded
    from config_db.json, so pre-existing routes (e.g. an operator's
    custom blackhole or VRF route) survived regen.  Clearing the owned
    tables now drops them before the management route is written.

Add tests pinning the model.  Illustrative helper tests
(test_config_generator_ownership.py) show that _add_vrf_configuration and
_add_vlan_configuration replace BGP_GLOBALS[vrf], VLAN entries and
ROUTE_REDISTRIBUTE keys wholesale.  Orchestrator tests
(test_config_generator_orchestrator.py) assert that pre-existing
BGP_GLOBALS['default'] fields, STATIC_ROUTE entries, and stale owned-table
entries are dropped on regen while DEVICE_METADATA and VERSIONS survive.

The ownership tests also pin the table-classification taxonomy: that the
scaffold set partitions into scaffolded-owned and inherited tables, that
owned and inherited tables are disjoint, that OWNED_TABLE_KEYS has no
duplicates, and that the scaffolded-owned and on-demand owned sets do not
overlap.  Most of the taxonomy holds by construction (OWNED_TABLE_KEYS is
derived), so these checks exist to fail when the hand-written
INHERITED_TABLE_KEYS or ON_DEMAND_OWNED_TABLE_KEYS literals are edited in
a way that would silently break the model.

AI-assisted: Claude Code
Signed-off-by: Roger Luethi <[email protected]>
@ideaship ideaship force-pushed the doc-sonic-generator-ownership-model branch from ba31c99 to bb263fa Compare May 29, 2026 09:06
The assert in test_owned_table_categories_are_disjoint fit on a single
line, so black flagged the manually wrapped form in CI.  Let black
collapse it.

AI-assisted: Claude Code
Signed-off-by: Roger Luethi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

3 participants