* plainbox:commands: show 'steps' information in CLI for manual jobs
This patch fixes a problem where plainbox would omit showing
'steps' field in CLI clients when running manual jobs. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1453067
* plainbox:commands: print field headers This patch makes CLI front-
ends print 'Purpose:', 'Steps:' and 'Verificaiton:' text before
printing contents of the corresponding fields. This should make
interaction with plainbox more digestable. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1453070
* plainbox:commands:inv_run: fix wrong reporting of estimated_time
This patch fixes reporting of time left in the session.
* plainbox:commands: add progress notification before starting each
job This patch makes plainbox report current progress (how many
jobs are done, out of how many, what's the estimated time to
complete the session). This notification is visible when using
both: plainbox run and checkbox with new-ui. Fixes:
https://bugs.launchpad.net/checkbox-ng/+bug/1392123
* plainbox:commands:inv_run: add seconds_to_human_duration funtion
This patch adds a function that converts number of seconds to
human readable string. It should be used when front-ends want to
display projected time left in a current testing session
* plainbox: add *.html from plainbox/data/ to MANIFEST.in Fixes:
https://bugs.launchpad.net/plainbox/+bug/1443472
* plainbox:box: load exporters for stubbox
* plainbox:docs: update examples using --output-format
* plainbox:exporter:hexr: export IO logs for just the attachments
The original XML exporter iterated over attachment map, we're
iterating over the job state map so we should filter out non
attachment jobs explicitly. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1464653
* plainbox:exporter:hexr: add additional tests
* plainbox:exporter:html: use correct loop index for re-runs
* checkbox-ng:service: log extra warnings when results cannot be
removed
* plainbox: ship hexr .xml template Fixes:
https://bugs.launchpad.net/plainbox/+bug/1463705
* plainbox:exporter:html: display result history
* plainbox:exporter:text: display result history
* plainbox:session:result: replay the full result history
* plainbox:session:suspend: store full result history
* plainbox:session:jobs: keep track of result history
* plainbox:result: remove all setters and change signals
* plainbox:exporter:hexr: fix escaping test The original test
injected "evil" data by setting the _io_log attribute (note the
leading underscore). This was easily mistaken since the original
code was accepting any attributes. I spotted this during the job
result builder conversion and decided to fix it properly.
* plainbox:exporter:hexr: use JobResultBuilder
* plainbox:commands:run: use JobResultBuilder This patch changes all
of the result handling in the implementation of "plainbox run" to
rely on job result builder. This is done to avoid any modification
of existing results and instead, if appropriate, construction of
new results. One minor detail is that the timing measurements are
now done at a lower level, where we still have the result builder.
* plainbox:runner: use JobResultBuilder everywhere This patch
changes all of the code to use JobResultBuilder. This change has
two implications. First we don't need to construct concrete result
types anymore so all of the code is more consistent. Second of
all, some private, helper methods now return a builder object,
instead of a result. This is done so that the caller can still
modify the result before finally returning a sealed result object
out of the JobRunner methods.
* plainbox:result: add _JobReturnBase.get_builder() To complement
the JobResultBuilder.get_result() we have
{Memory,Disk}JobResult.get_builder(). During the current
transition it might be helpful to construct and de-construct
result objects (as we get new objects all the time we don't
trigger any modification warnings).
* plainbox:result: add JobResultBuilder This patch adds a new class
that assists in building appropriate JobResult objects. The result
builder has a .get_result() method that returns a regular result
(of the appropriate kind, either disk or memory). The idea is to
make the result immutable later on and only allow modification to
happen inside the builder.
* plainbox:exporter:html: remove redundant calls to
get_certification_status_map()
* plainbox:exporter:html: remove commented-out css rule
* plainbox:exporter:html: don't use deprecated -vendor properties
* plainbox:exporter:html: fix some inconsistent css syntax
* plainbox:exporter:html: remove unused css rules
* plainbox:exporter:html: inline checkbox.js
* plainbox:exporter:html: inline checkbox.css
* plainbox:exporter:html: use a variable for job_state_map
* plainbox:exporter:html: use a variable for resource_map
* plainbox:exporter:html: don't spell the long namespace over and
over
* plainbox:exporter:html: remove useless newlines
* plainbox:exporter:html: remove useless OOM object
* plainbox:exporter:html: fix broken substitution on tr_outcome
* plainbox:exporter:html: re-factor construction of test data This
patch replaces the large mocking constructs that provided test
data for the html exporter with real objects. It is actually much
shorter and easier to set it up this way.
* plainbox:exporter: add HEXR exporter This patch adds a new
exporter (HEXR) that is intended to replace our current "xml"
exporter. The schema is specific to the Canonical HEXR application
so the name reflects that. Note that due to the syntax used, this
exporter requires Jinja2 >= 2.7. The dependency wasn't there
before so I've added it to install_requires.
* plainbox:exporter:jinja: add support for customizing Environment
The environment is how you control all of jinja's behavior.
Currently the environment is hidden in the initializer of the base
jinja exporter class. By having a method where we can customize
the environment _before_ a template is loaded we can add anything
(additional tests and filters). After a template is initialized,
those changes don't affect the template.
* plainbox:providers:special: allow passing keyword args to
get_stubbox() The provider constructor seems to call the old
validate API. Since this is pretty annoying (all the incorrect
things the old validation says) I wanted to have a way to disable
that by passing validate=False, check=True. This is a quick stop-
gap measure. In reality we want to purge all the old validation
code.
* plainbox:stubbox: add set of representative jobs This patch adds a
number of jobs that validate and don't do anything. Each job has
an identifier that looks like "representative/plugin/{plugin}".
The goal of this collection is to simplify unit tests that
traditionally re-created everything using python APIs (which is
both fragile and more verbose).
* plainbox:resource: make it possible to iterate over resources
Iterating over resources behaves just like iterating over
dictionaries (keys are returned). This is useful to access
resource elements in Jinja templates. In the past we just accessed
._data directly but this is a hack and we should get rid of it.
* plainbox:commands:check_config: produce more useful output This
patch changes the 'plainbox check-config' command so that it is
immediately obvious which ini-file section each variable belongs
to. I found that I didn't remember some settings and I had to go
and chase the code to see the definitions. With this tweak, each
variable looks like a part of the actual ini file so everything is
obvious.
* plainbox:pod: add Field.change_notifier This patch adds a new
decorator that can be used similarly to @foo.setter (when foo is a
@property-decorated method). This can simplify all fields that
need to have a custom notification method. All such methods can be
implemented _after_ all of the field definitions now.
* plainbox:pod: filter-out UNSET from as_dict() Normally UNSET
values don't exist in a POD for a long time. Typically they are
only used briefly during POD initialization. If the developer uses
a custom default value and actually uses UNSET (or assigns UNSET
explicitly) then as_dict() will misbehave, producing the UNSET
value again (which is implicit). This patch changes as_dict() so
that it doesn't propagate them to the dict representation. Note
that this is different from the as_tuple(), which cannot avoid
representing UNSET.
* plainbox:pod: add unset variants for type assignment filters This
patch adds pod.unset_or_typed and pod.unset_or_typed.sequence
assign filters. Those two new filters allow to use UNSET values
(when explicitly desired) and still get a benefit from the type
checks that typed and typed.sequence offer. UNSET acts as a kind
of variant of None, offering nullable types. This is not the best
way to do it but I wanted to have a distinction between None and
UNSET so that None can be used normally.
* plainbox:pod: fix copy-pasted docstring
* plainbox:pod: fix sequence_type_check_assign_filter() This patch
fixes a silly issue in sequence_type_check_assign_filter() that
wasn't tested. The raised exception should have mentioned the type
of the sequence elements, not the type of the actual sequence.
Yes, tests find bugs. Untested code is almost always broken.
* plainbox: use @morris.signal instead of @signal This patch changes
all of the code that looks like @signal def blah(...): ... To look
like this instead: @morris.signal def blah(...): ... This prevents
us from shadowing the stdlib signal module by accident. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1442175
* plainbox:resume: enable 5th resume helper
* plainbox:resume: split tests for DiskJobResult 1-4 and 5 This
patch splits unit tests for DiskJobResult in formats 1 through 4
and in the 5th format. Tests for the 5th format check the extra
conditions and behavior related to relative and absolute pathnames
* plainbox:resume: add 5th resume helper
* plainbox:resume: use absolute path for DiskJobResult resume test
An absolute path is more realistic. Using it will also help with
the upcoming 5th format.
* plainbox:resume: add SessionResumeHelper1._load_io_log_filename()
This patch moves a part of SessionResumeHelper1._build_JobResult()
method to a new, tiny method so that it can be overriden
separately for 5th resume format.
* plainbox:resume: enable extra testing for 4th format
* plainbox:resume: make SessionResumeHelper1.__init__(...) arguments
mandatory This patch makes the flags and location arguments of
SessionResumeHelper1 mandatory in anticipation of the fifth resume
format that relies on a sane value of "location"
* plainbox:resume: better docstring for
SessionResumeHelper1.__init__()
* plainbox:resume: better docstring for
SessionResumeHelper.__init__()
* plainbox:resume: make _build_JobResult(flags, location) mandatory
This patch makes all of the arguments to _build_JobResult()
mandatory.
* plainbox:resume: pass flags and location to _build_JobResult()
This patch modifies all calls to _build_JobResult() so that flags
and location arguments are provided. This will allow to make them
mandatory later. The intent is to make them mandatory as they will
be required for 5th resume format and all non-test code already
uses them.
* plainbox:resume: make SessionResumeHelper.__init__() args
mandatory This match makes it mandatory to pass the location and
flags arguments to the session resume helper. This is done so that
the location argument is not omitted by accident anywhere.
Location will be mandatory soon, with the 5th resume format.
* plainbox:resume: fix PEP-8 issue
* plainbox:resume: fix typo
* plainbox:manager: pass 2nd argument to suspend() This patch makes
the session manager pass the session_dir hint argument to the
suspend helper. This way the helper can optimize the
representation of the session to that directory.
* plainbox:suspend: switch to 5th format by default
* plainbox:suspend: add 5th suspend format This patch adds the fifth
suspend format. This format is the same as the fourth format with
the exception of how DiskJobResult is saved. If the new (but still
optional) session_dir argument is provided then a relative
pathname to each DiskJobResult is saved instead of an absolute
one. This allows the whole session to be relocated easily. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1378295
* plainbox:suspend: use a more convenient io_log_pathname
* plainbox:session: decouple helper class from tests
* plainbox:suspend: push session_dir to all helper methods This
patch pushes the session_dir argument to all of the helper methods
on all of the SessionSuspendHelper1 subclasses.
* plainbox:suspend: add session_dir argument to suspend() This patch
adds a new optional argument to SessionSuspendHelper1.suspend().
The optional argument can be used to tell the suspend helper where
the data will be stored. This can be used to use relative
pathnames inside the representation.
* plainbox:pod: add @podify decorator This patch adds a new
decorator, @podify, that converts any class that inherits from
PODBase to a POD-lookalike. This way one can make PODs out of
classes with custom metaclasses as well as use POD-based
exceptions.
* plainbox:pod: move POD methods to new PODBase class This patch
moves all of the methods unique to a POD to the new PODBase class.
This class has no metaclass and can be inherited by anything that
wants to do so.
* plainbox:pod: use field.counter to determine field ordering This
patch changes _FieldCollection.inspect_namespace() to use
field.counter to detect ordering of Field() entries as they appear
in the source code. This will make it possible to drop the
requirement to use the __prepare__() method later.
* plainbox:pod: add .counter to each Field. This patch adds a ever-
incrementing counter to each Field instance. This field can be
used to look at the order in which particular fields were defined
when we cannot control the namespace in which a class is defined
in (via the __prepare__ metaclass method)
* plainbox:pod: fix PEP-8 issues
* plainbox:pod: fix PEP257 issues
* providers:checkbox: don't use dbus-launch Using dbus-launch is
always wrong, as explained in the bug linked below. Fixes:
https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1377266
* plainbox:commands:run: don't crash when job has unknown duration
This patch fixes a crash that happens when a job has unknown
estimated duration. It also ensures that the UI looks useful when
that happens by changing the estimate to "unknown". Fixes:
https://bugs.launchpad.net/plainbox/+bug/1445030
* plainbox:depmgr: add extra debugging logs
* plainbox:depmgr: replace integers with Color enumeration values
* plainbox:depmgr: add Color enumeration This patch adds a classic
three-value color enumeration for graph traversal algorithms.
* plainbox:depmgr: fix PEP257 issues and improve docstrings. Most of
the changes are related to ensuring proper spacing and
punctuation. Some changes are re-wording docstrings for
consistency among similar classes.
* plainbox:unit:job: materialize imports before passing to resource
program This patch fixes an interesting bug where a "imports" line
would only work for the first out of all the requirements. The
reason for this is that the implementation of the imports system
uses a generator and the resource program iterates over that
iterator more than once. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1444242
* plainbox:public: add get_providers This patch exposes
get_providers() as the first public method (apart from main).
* plainbox.public: load functions lazily This patch replaces the old
'@public' decorator with LazyModule.
* plainbox:lazymod: add the _lazymod module This patch adds a copy
of code from python-glibc (that I also wrote) that simplifies
creation of 'lazy modules'. Those modules can conveniently expose
various objects (functions, classes, anything really) and only
load it on demand. I intend to use it for the public module to
work around some circular import issues that were present with the
older approach that did all of the imports immediately.
* plainbox:providers: add get_providers() function This patch adds a
new (public) function that addresses the problem of confusing APIs
for loading all available providers. Unlike past approaches this
is a simple function, it's not a class nor an object with complex
API. This should make it easier to understand and use within
applications.
* plainbox:providers: remove stray newline from the license
* plainbox:ctrl: skip non-jobs printed by local jobs This patch
makes the session state controller skip non-job units that are
printed by local jobs. In theory we could add more units this way
but the code for handling that is ugly as it is, without the
holding area, so this is the better compromise. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1443228
* plainbox:ctrl: use real RFC822 record for testing This patch
changes some tests to use a real RFC822 record for testing session
controller code. The intent is to let that real record behave
correctly when upcoming code inspects it for some data.
* providers:manifest: add use-chunked-io flag to collect-manifest
job
* plainbox:commands:run: flush on each part of IO This patch enables
explicit flush on each write that came from a job process. This is
normally line-based IO for all the text-based jobs. When using
CHUNKED_IO with legacy thread+blocking IO extcmd this is actually
triggered on each byte read so we get "live" output, as it
happens.
* plainbox:commands:run: preserve original line endings for display
* plainbox:runner: use CHUNKED_IO for jobs that request it This
patch enables CHUNKED_IO mode for jobs that have the 'use-chunked-
io' flag.
* plainbox:vendor:extcmd: use 1-byte buffers for legacy chunked IO
This patch is a hack to get to an useful effect from legacy
(thread based) chunked IO implementation. By default chunked mode
used a 4K buffer which was a reasonable value for bulk data
transfers. Since the blocking API doesn't allow us to see anything
before the whole buffer gets full we cannot use it to get things
like interactive prompts to work. In order to fix bug
https://bugs.launchpad.net/plainbox/+bug/1443911 I wanted to use
chunked IO (and it works great in glibc mode which uses non-
blocking IO). To use chunked IO in thread + blocking mode we can
cheat and use 1-byte buffers. That is, each byte that gets written
to a pipe gets read and reported to the application. This is
slower and by far, but it gets the job done. Since chunked IO is
enabled selectively and so far only the manifest job enables it,
it's a safe hack / compromise.
* plainbox:vendor:extcmd: add support for chunked I/O This patch
adds a way to do non-line-based I/O via extcmd. Line base I/O is
perfect for text but can be problematic and inefficient for binary
data. By allowing applications to pass the new CHUNKED_IO flag to
extcmd, they can now opt-into the new behavior (per extcmd
instance) which calls the new on_chunk() delegate method, instead
of the old on_line() method.
* plainbox:depmgr: don't crash when directly solving unknown jobs
This patch changes the DependencyManager class not to crash when
asked to solve the dependency chain of a given job, directly (as
if it was placed on the desired_job_list) but the job is not
present in the solver copy of job_list. Instead of raising
KeyError, a new error (subclass of DependencyError)
DependencyUnknownError is raised. This error is handled uniformly
by the SessionState class so everything works as expected. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1334296
* plainbox:state: add fix and regression test for 1444126 This patch
fixes a bug in SessionState.update_desired_job_list() that could
cause a job to be removed twice (causing ValueError to be raised).
As explained in the associated regression test, the problem is
caused by the following chain of dependencies: A -> B -> C The
session knows about jobs A and B, C is unknown. We want A and B
The solver loop results in the following actions: - (A depends on
B so we go to B and notice that it depends on C). - B gets removed
from desired_job_list because C is unknown
(DependencyMissingError). - The problem is recorded and we try
again (another solution, this time just looking for [A]) - (A
depends on B so we go to B and notice that it depends on C) -
Crash, B was already removed! The fix also involves removing jobs
from the set of known jobs because otherwise an endless loop
occurs: - (A depends on B so we go to B and notice that it depends
on C). - B gets removed from desired_job_list because C is unknown
(DependencyMissingError). - The problem is recorded and we try
again (another solution, this time just looking for [A]) - (A
depends on B so we go to B and notice that it depends on C) - B
was already removed so we don't do anything and we try again
(oops!, nothing has changed we'll be here forever) Fixes:
https://bugs.launchpad.net/plainbox/+bug/1444126
* plainbox:depmgr: add DependencyMissingError.__hash__()
* plainbox:depmgr: add DependencyMissingError.__eq__()
* checkbox-ng: load manifest provider This patch makes checkbox-ng
use extra APIs to load the manifest provider. This is not the best
way to solve it as plainbox should really just load all of the
special providers as well but for now this will have to do. There
will be separate patches to fix the provider API next. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1443895
* plainbox:testplan: don't crash if id is not set This patch
prevents a crash that can happen when a test plan doesn't have an
ID field. This is normally validated separately but the context
validator (which does other checks) should be written defensively
and not assume that each unit is correct. Here the problem was
related to trying to come up with a regular expression match of a
None id against some unrelated test plan pattern. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1443899
* checkbox-ng:ui: don't crash if there are no nodes to toggle This
patch fixes the 'toggle' feature where the test operator can press
enter to expand/collapse a test node. Since having no nodes at all
is possible (though not common) it's required to check if we got a
node in the first place before going to toggle it's state. Fixes:
https://bugs.launchpad.net/checkbox-ng/+bug/1443901
* plainbox:manifest: correct location of the hardware manifest This
patch adds the missing 'share/' directory part to closer mimic the
XDG_DATA_HOME directory. Fixes:
https://bugs.launchpad.net/plainbox/+bug/1443903
* plainbox:ctrl: detect and break via cycles This patch changes the
CheckBoxSessionStateController to pay extra attention not to
create via cycles when processing local jobs. Fixes: https://bugs.
launchpad.net/plainbox/+bug/1441993https://bugs.launchpad.net/plai
nbox/+bug/1441993
* plainbox:ctrl: add get_via_cycle() This patch adds a function that
detects cycles in via connections.
* plainbox:ctrl: rewrite tests for local jobs This patch rewrites
three tests for local jobs, not changing their semantics but
changing the implementation. The original implementation used
mocking for everything and focused on observing impact of the
session controller on the mocked session. This turned out to be
difficult to adapt to changes to the controller so instead non-
mocked objects are used and minimal mocking is used as a short-cut
(to remove the necessity to create real results that correctly
define jobs).
* plainbox:exporter: remove unnecessary xml import in html exporter
* Fix missing info in the HTML report (requirements and blocker
sections)
* plainbox: remove dependencies on lxml This patch removes all of
the remaining mentions of lxml.
* plainbox:exporter:xml: Remove the old XML exporter module Remove
dependency and references to lxml
* plainbox:data:report: Remove the hexr/html templates from
plainbox/data They are now the PROVIDER_DATA directory of the
exporter special provider.
* plainbox:exporter: Remove old html/hexr exporters Those are now
entirely generated using the Jinja2 exporter
* plainbox:secure:config: Allow keys w/o values and set '=' as
unique delimiter
* plainbox:setup.py: Use the HEXR exporter to generate XML
* plainbox:commands:run: Load exporters from their units `plainbox
-f` now only accept valid exporter unit ids. The exporter is
created after we get a session to query the manager and get all
exporter units
* plainbox:exporter: Update exporters __init__() with an
exporter_unit argument All exporters get a new argument called
exporter_unit meant to store the new ExporterUnitSupport.
Exporters can now have access to the custom data property and
tweak the reports in many ways. Options are still a positional
argument but can be extended by options set in the data field of
the exporter unit, in the "extra_options" key. The Jinja2 exporter
adds to the list of paths where to look for templates the special
PROVIDER_DATA directory of the exporter unit. Providers can now
ship new templates for Jinja2 based exporters (HTML is obviously a
perfect candicate) Both the HTML and HEXR exporter classes are now
merged into their base Jinja2 exporter class.
* plainbox:exporter:text: Fix Colorizer import
* plainbox:session:manager: Add the exporter_map property A Map from
exporter id to the corresponding exporter support unit. This is
the preferred way to get such units as the manager knows all the
loaded providers and their related units. Note: Only units passing
the validation check() are kept.
* plainbox:providers: Special provider for existing exporters The
xlsxwriter package is now mandatory (at least for development)
* plainbox:unit:exporter: New unit type allowing exporter
customization The mandatory fields are the unit id and the
setuptools entry point to call. The exporter summary will be used
by the `plainbox run -f ?` command. The exporter_cls method is the
target of the entry point. If plainbox cannot load the entry point
it will exit automatically. The data field is a special optional
field meant to be used to pass a complex object to the exporter.
As RFC822 was not suitable for such container, the data field must
be in JSON format. E.g: data: { "foo": "bar", "baz": [ 1, 2, 3] }
* plainbox:docs:manpages: New man page for exporter units
