Learning

For 7 days after installation, and 48 hours for each new device, Warda only observes: warda status already shows the devices, what was blocked for each one and how many suspicious names are under observation. The 7 days are counted in running time: a box switched off observes nothing, so the period may last more days of the calendar than it counts (see The observation period). Then, every hour, it scores the names seen over the last 7 days:

• a keyword in the name (telemetry, analytics, tracking… or, with other names of the same domain already blocked, stats, events…) is required;
• calls in the background, at regular intervals, at night, or from several devices add confidence.

The level sets how many suggestions you get: low (fewest, most certain), medium (default) or high. A domain you reject is never suggested again; a suggestion you accept then unblock counts as an error of the scorer. Your own domains (WARDA_OWN_DOMAINS), the machines that work alone (warda device -kind server; from 0.7.10 also the kinds NAS, firewall or router, access point or switch, see Kinds of devices) and a built-in list of vital services (system updates, banks, public services, consoles, streaming…) are never suggested. A suggestion found on a device of an adult who did not agree to detailed reports (or of a group their account made) gives neither the device nor the site visited just before ("private": true in GET /api/v1/suggestions).

Suggestions → Watched names (the tab was named "Under observation" before 0.6.8) shows the suspicious names of the last analysis that are not suggested yet, the highest confidence first, each with its role, its devices, its confidence and why it waits:

• observing: Warda is still in its first observation period (with its end date);
• below_threshold: its confidence is below the threshold of the current level (with the level from which it would be suggested, or none);
• device_warmup: only new devices asked it, still observed (with the end of their observation);
• server: only devices marked as servers asked it (from 0.7.10: only machines that work alone, of the kinds server, NAS, firewall or router, access point or switch);
• weekly_cap: the suggestions of the week are all made.

A status card gives the level and its threshold, the last analysis, the number of suspicious names and the end of the observation; Analyse now (the roles that change the rules) runs the analysis at once. With the learning off, the page says so, with a link to the settings. The empty view of the suggestions waiting links to this tab. Nothing watched is blocked.

The card What Warda is watching, at the top of the tab, tells it in plain words for a household: how many names Warda watches because they may serve to follow or measure what is done on the network, that none is blocked, and that a name becomes a suggestion once Warda is sure enough. It gives the threshold of the level chosen ("At your level (Medium), a name is suggested once Warda is 70 % sure") and how many names each level would suggest with what Warda knows now. The thresholds are those of the learning: Low 90 %, Medium 70 %, High 50 % (a level suggests every name at or above its threshold: High suggests the most). The chart puts each name as a dot on a strip of confidence from 0 to 100 % (the names of close scores stacked), with a line for the threshold of each level and the level of the box highlighted; its legend reads "A name watched", "Threshold of a level" and "Suggested at your level", and a text alternative gives the same figures to a screen reader. Level of detail switches between Simple (the default) and Detailed (remembered in the browser), which adds the counts by reason held ("Why they are not suggested yet": below the threshold, observation period, new device, servers only, weekly limit, each explained) and by role ("What they probably do"). Everything is computed in the browser from GET /api/v1/suggestions/observed (names[].score, held, role, thresholds).

GET /api/v1/suggestions/observed gives learning (level, observing, observing_until, last_run, candidates; from 0.7.10 observed_days, observation_days, observation_ended, names, devices, window_days, suggested, see The observation period), threshold, thresholds (by level), per_week and names (domain, role, role_text, score, level, devices, device_label, evidence, held, until; private as for the suggestions). POST /api/v1/learn (and Analyse now) answers 409 while an analysis runs or when the last one ended less than a minute ago.

The Learning card of the dashboard follows the first observation week: during it, the end of the observation, the day reached (seven marks) and the level; with the learning off, it says so. From 0.7.10 it then tells, for a week, what the observation gave (below), and disappears from the dashboard (and from the modules Customize can add) a week after the end of the period — from 0.6.9 to 0.7.9 it disappeared as soon as the 7 days were over. The learning goes on (every hour): its suggestions are on Suggestions, and Suggestions → Watched names is the place to judge it. With the collective base on, the watched names and the suggestions show the opinion of the analysis service of Warda when one is kept ("Opinion of Warda: suspicious — registered 3 days ago, listed by …").

The observation period#

From 0.7.10 (internal/learn). What was wrong before: the day of the period and its end were those the analysis of the hour had seen, so the dashboard could say "day 8 of 7", or "observing until" a time already gone, up to an hour after the end, and nothing told that the period had ended nor what it had given.

• Counted in running time. The period is 7 days of Warda running (observed_ms, added to every 5 minutes and at the stop, on the monotonic clock: a change of the date does not count). A box switched off for a night observes nothing meanwhile: the period then lasts more days of the calendar than it counts, and the end announced moves that much later.
• The day never passes the number of days. observed_days goes from 1 to observation_days (7), never more, and is read as it is now, not as the last analysis saw it ("Day 3 of 7" on the card).
• The end really happens, at its hour. As soon as the period is over an analysis starts, without waiting for the next hour: the first suggestions are made then. The end is kept (observation_ended, in the backups; own to each box of a pair) and logged (observation period over: the suggestions start).
• The end is told, with what was studied. For a week after the end, the card of the dashboard says "The observation of your network ended on …: Warda now makes its suggestions.", with the names studied at the last analysis, the devices that asked them, the names watched and the suggestions made, then either what comes next ("Warda goes on studying the journal every hour: a watched name becomes a suggestion once Warda is sure enough.") or, when nothing was suggested, "Nothing needed a suggestion: no name asked on your network looks like it serves to follow or measure without the lists blocking it already.", and that the card leaves the dashboard a week after the end. Suggestions keeps it afterwards: Names studied at the last analysis, Devices that asked them, Suggestions made since the installation (waiting, blocked, kept allowed), Observation period ended on, and what happens next; an empty page says what was studied instead of looking dead.
• Kept across restarts. The last analysis and its watched names are kept in the database (learning of the table meta) and shown again after a restart, until the next analysis (for the same level only); a restored backup starts from the next analysis.

learning of GET /api/v1/suggestions/observed and of the summary of the dashboard: observed_days and observation_days (the day reached and the number of days), observation_ended (absent while the period lasts), names and devices (the names answered over window_days days that the last analysis studied, the devices that asked them), suggested (the suggestions the learning made since the installation, by status, as they are now: pending, accepted, rejected, reverted).

Automatic blocking of confident trackers#

From 0.8.0 (free, beta, every plan; internal/admin/learnauto.go, hook OnSuggested of internal/learn). The threat it answers: the telemetry of connected objects and TVs, which keeps calling new names that no list knows yet, while nobody reads the suggestions of the box. Until 0.7.13 the learning never blocked anything by itself; it still does not unless an administrator asks for it.

Suggestions, card Block confident new trackers automatically (below the tabs; the roles that change the rules): off by default, and it can be turned on only at the learning level High (set in the settings). If the level is lowered later, the card says "On, but paused" and nothing is blocked automatically meanwhile.

What is blocked: a suggestion of the local learning (never one of the collective base) whose role is tracking or telemetry (analytics is left out: more often the first-party statistics of the site visited, whose blocking breaks pages), whose score reaches 0.9 (the threshold of the level Low, the surest), and made after the option was turned on (new trackers only). Never: a name of protect.txt or of WARDA_OWN_DOMAINS; a name (or a parent) that has a rule already, an administrator's allow, a configured allow or an allow of the lists; a known site or a site the household visits most (the names the look-alike protection protects, and the top names of the household); a name undone before. At most the weekly number of suggestions of the learning (PerWeek) over 7 days.

Each one becomes a rule of the source learn-auto (shown on My rules as "Blocked automatically (learning)"), and its suggestion is marked accepted. It runs right after each analysis that made suggestions (every hour, or Analyse now). Undo (on the card, or deleting the rule on the rules page) removes the rule, marks the suggestion reverted (an error of the scorer, as for a suggestion accepted then unblocked) and puts the name on a list of names never blocked automatically again (2,000 at most). Each run that blocked something leaves an alert learn.auto for the administrators ("Warda blocked by itself N new trackers it is sure of: a, b, c… Each one can be undone on the Suggestions page", up to 5 names; group Security of Alerts, SIEM severity 2). The summary of the week is not changed.

API: GET /api/v1/suggestions/auto (every account) gives enabled, available (the level is High), level, since, threshold, roles, per_week, week (blocked over 7 days), blocked (domain, role, score, created) and never; PUT /api/v1/suggestions/auto {"enabled": true} (rules written; 409 unless the level is High; audit learn.auto.on/off); POST /api/v1/suggestions/auto/{domain}/undo (rules written; 404 for a rule it did not make). Audit suggestion.auto for the names blocked. Kept in meta (learn_auto: when it was turned on; learn_auto_never; learn_auto_log: the blocks of 30 days); from 0.8.2 in the backups and given to the secondary of a pair (see Backups; before, a restore kept the choice of the box it was restored on; a new box has it off), and the rules it made, which are rules, come back with the backup. The secondary of a pair never blocks by itself: its rules come from the primary. Nothing leaves the box.

Limits: a score is a probability, not a proof; an automatic block may break a function of an object (its update, its remote control): undo it, and the name is never blocked automatically again.