sonic: enforce full config_db.json ownership#2297
Open
ideaship wants to merge 2 commits into
Open
Conversation
cae57c6 to
9952d93
Compare
9952d93 to
f4c34e0
Compare
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The ownership model is currently encoded via INHERITED_TABLE_KEYS + OWNED_TABLE_KEYS and duplicated table-name literals; consider centralizing these names (e.g., a single authoritative enum/set) and/or adding a small invariant check so new scaffold keys or ownership changes can’t silently drift from the documented model.
- Given that OWNED_TABLE_KEYS is used to pop sections before scaffolding, it may be worth explicitly asserting in code (or a small helper) that every TOP_LEVEL_SCAFFOLD_KEY other than INHERITED_TABLE_KEYS is indeed in OWNED_TABLE_KEYS, so future additions don’t accidentally become neither owned nor inherited.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The ownership model is currently encoded via INHERITED_TABLE_KEYS + OWNED_TABLE_KEYS and duplicated table-name literals; consider centralizing these names (e.g., a single authoritative enum/set) and/or adding a small invariant check so new scaffold keys or ownership changes can’t silently drift from the documented model.
- Given that OWNED_TABLE_KEYS is used to pop sections before scaffolding, it may be worth explicitly asserting in code (or a small helper) that every TOP_LEVEL_SCAFFOLD_KEY other than INHERITED_TABLE_KEYS is indeed in OWNED_TABLE_KEYS, so future additions don’t accidentally become neither owned nor inherited.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
f4c34e0 to
ba31c99
Compare
Contributor
Author
|
Updated with constant split + taxonomy invariant tests. No functional change. |
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The ownership taxonomy (INHERITED_TABLE_KEYS, SCAFFOLDED_OWNED_TABLE_KEYS, ON_DEMAND_OWNED_TABLE_KEYS) is currently maintained by hand; consider centralizing the table names in a single source (e.g., a mapping from table → classification) so that adding/removing tables in helpers can’t silently fall out of sync with these constants and the docstring.
- The generate_sonic_config docstring hard-codes several table names (BGP_GLOBALS, VLAN, VLAN_INTERFACE, etc.) that are also captured in OWNED_TABLE_KEYS; to avoid drift, it may be cleaner to either reference OWNED_TABLE_KEYS symbolically in the docstring or reduce the explicit list there and rely on the constants for the authoritative set.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The ownership taxonomy (INHERITED_TABLE_KEYS, SCAFFOLDED_OWNED_TABLE_KEYS, ON_DEMAND_OWNED_TABLE_KEYS) is currently maintained by hand; consider centralizing the table names in a single source (e.g., a mapping from table → classification) so that adding/removing tables in helpers can’t silently fall out of sync with these constants and the docstring.
- The generate_sonic_config docstring hard-codes several table names (BGP_GLOBALS, VLAN, VLAN_INTERFACE, etc.) that are also captured in OWNED_TABLE_KEYS; to avoid drift, it may be cleaner to either reference OWNED_TABLE_KEYS symbolically in the docstring or reduce the explicit list there and rely on the constants for the authoritative set.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
The SONiC config generator owns config_db.json in full: it generates the file from NetBox data and hardcoded SONiC policy, and operators must not make manual adjustments to it. This ownership model was implicit and undocumented, so each PR touching the generator relitigated which sections it owns (surfaced in review of #2279 and #2290). Document the model in the generate_sonic_config() docstring as whole-file ownership: the generator overwrites every section it manages, and no sections pass operator edits through unchanged. Note the two fields the generator does not itself emit and instead inherits from the device's image-provided base config (the DEVICE_METADATA localhost device attributes and the DATABASE schema VERSION); these come from the image, not from operator hand-edits. Bring the code into line with the documented model: - Rebuild owned tables from scratch instead of overwriting only the entries touched in the current run. generate_sonic_config() deep- copies the base config and most helpers assign only the keys they currently generate, so an entry removed from NetBox but still present in config_db.json (e.g. an old VLAN_MEMBER, BGP_GLOBALS VRF, or VXLAN_TUNNEL_MAP) survived regen as stale, active config. Drop the content of every owned table up front and only regenerated entries remain. OWNED_TABLE_KEYS is composed from two named sub-categories: the scaffolded-owned tables (every scaffold key except the inherited DEVICE_METADATA and VERSIONS, created up front via setdefault) and the on-demand owned tables (ROUTE_REDISTRIBUTE, the SNMP_* tables and SYSLOG_SERVER, which helpers assign only when NetBox carries their data). - BGP_GLOBALS['default'] was written with key-level updates that merged into any pre-existing entry, so operator fields survived regen. Replace the entry wholesale when a primary IP is present, like every other generated VRF. - STATIC_ROUTE merged the management default route into the dict loaded from config_db.json, so pre-existing routes (e.g. an operator's custom blackhole or VRF route) survived regen. Clearing the owned tables now drops them before the management route is written. Add tests pinning the model. Illustrative helper tests (test_config_generator_ownership.py) show that _add_vrf_configuration and _add_vlan_configuration replace BGP_GLOBALS[vrf], VLAN entries and ROUTE_REDISTRIBUTE keys wholesale. Orchestrator tests (test_config_generator_orchestrator.py) assert that pre-existing BGP_GLOBALS['default'] fields, STATIC_ROUTE entries, and stale owned-table entries are dropped on regen while DEVICE_METADATA and VERSIONS survive. The ownership tests also pin the table-classification taxonomy: that the scaffold set partitions into scaffolded-owned and inherited tables, that owned and inherited tables are disjoint, that OWNED_TABLE_KEYS has no duplicates, and that the scaffolded-owned and on-demand owned sets do not overlap. Most of the taxonomy holds by construction (OWNED_TABLE_KEYS is derived), so these checks exist to fail when the hand-written INHERITED_TABLE_KEYS or ON_DEMAND_OWNED_TABLE_KEYS literals are edited in a way that would silently break the model. AI-assisted: Claude Code Signed-off-by: Roger Luethi <[email protected]>
ba31c99 to
bb263fa
Compare
The assert in test_owned_table_categories_are_disjoint fit on a single line, so black flagged the manually wrapped form in CI. Let black collapse it. AI-assisted: Claude Code Signed-off-by: Roger Luethi <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Regenerating a SONiC config from NetBox repeatedly raised the same question: when
generate_sonic_config()rewritesconfig_db.json, which sections does it own, and may an operator hand-edit anything directly? PRs #2279 and #2290 both ran into it. The policy is now settled: the generator owns and generatesconfig_db.jsonin full, and operators make no manual adjustments to it.What this PR does
Documents that ownership model in the
generate_sonic_config()docstring and brings the code in line with it.DEVICE_METADATAlocalhost device attributes (e.g. devicetype) and theDATABASEschemaVERSION— are inherited from the device's image-provided base config, not from operator hand-edits.config_db.json(e.g. an oldVLAN_MEMBER,BGP_GLOBALSVRF,VXLAN_TUNNEL_MAP, SNMP user, or syslog host) survived as stale, active config. The owned tables (OWNED_TABLE_KEYS— every generated table except the inheritedDEVICE_METADATAandVERSIONS) are now cleared up front, so only regenerated entries remain.BGP_GLOBALS['default']replaced wholesale. It was written with key-level updates that merged into a pre-existing entry, so operator fields survived regen. It is now replaced wholesale, like every other generated VRF.STATIC_ROUTEno longer preserves hand-edited routes. Clearing the owned tables drops pre-existing routes (e.g. an operator's custom blackhole or VRF route) before the management default route is written.Tests
test_config_generator_ownership.py):_add_vrf_configurationand_add_vlan_configurationreplaceBGP_GLOBALS[vrf],VLANentries, andROUTE_REDISTRIBUTEkeys wholesale.test_config_generator_orchestrator.py): pre-existingBGP_GLOBALS['default']fields,STATIC_ROUTEentries, and stale owned-table entries are dropped on regen, whileDEVICE_METADATAandVERSIONSsurvive.