[ubuntu-hardened] Making security notices searchable

[Steve Beattie]

Hey Dominik,

Thanks for chiming in.

On Mon, Oct 14, 2019 at 04:00:26PM +0200, Russenberger Dominik wrote:
> Yes, a search would be great. My current usecase: we have an internal,
> partial mirror of the Ubuntu repos (multiple releases, different package
> set for each release). I have to check the repos for USNs for packages
> in there. My current solution is very... clumsy.
>
> On Sun, Oct 13, 2019 at 10:55:45AM +0100, Matthew Paul Thomas wrote:
> > As part of this, I???m considering adding a search function, so that you
> > can search security notices by Details text, with filters for Release
> > and Package.
> Filtering by arches (e.g. amd64+all) might be useful, for the handful of
> bugs that only affect a single architecture.

That might be possible but I don't think the result is going to be
satisfactory, in that architecture specific vulnerabilities are the
exception and not the rule.

> > So, I???m interested in knowing:
> >
> > *   Is there any current method of searching USNs? (Other than using
> >     ???site:usn.ubuntu.com??? with a global search engine, or grepping the
> >     usn.ubuntu.com Git repo.)
> Well... my current solution is to get https://usn.ubuntu.com/releases/ubuntu-18.04-lts/
> and then parse the HTML. Atom/RSS also just contain HTML for the content,
> but no method to filter by release. And of course parsing HTML  breaks 
> every once in a while.
> 
> > *   If any search showed results sorted newest first, would there be any
> >     use case for searching notices by date? (For example, show me only
> >     notices posted in 2017.)
> My usecase requires to show all USNs after a particular date, or even 
> better all USNs after another USN.

> > *   Anything else you think I should know?
> Machine-readable USNs! Alex Murray posted the link to a big json, but
> downloading&parsing 130mb every hour doesn't sound like such a good
> idea.

Understood. There is a compressed versions of the json blob available.
https://usn.ubuntu.com/usn-db/database.json.bz2 which is currently
"only" ~20MB in size, which may help that. Also, you shouldn't need
to re-download it every hour, a time check to see if it's newer
should suffice.

That said, one of the issues that makes the USN json db explode in
size is that it includes a reference for every binary package URL for
each USN; for something like a linux kernel update, this can result
in a couple of thousand URLs per USN, including things like udebs
that are only used by installer media. I'd like to provide a slightly
minimized USN json db that doesn't not contain those (but would still
contain the relevant source and binary packages). I've not yet done
an experiment to see what the size reduction would be, but I bet it's
significant and would reduce the amount of data to be consumed.

Would it also be useful to split the USN json db into per release
files as well?

Would a more digestible sized json db reduce the needs for complex
queries of the website itself?

Thanks again!

-- 
Steve Beattie
<sbeattie at ubuntu.com>
http://NxNW.org/~steve/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-hardened/attachments/20191014/eac9efcc/attachment.sig>