The WebSed processor scrapes fields from a wiki page or web site page, and substitutes field values into a text that is emitted by the processor. This can be used to scrape web sites for dynamic data (like stock quotes), or to scrape web files for test results data, or do a 'madlibs' style replacement on a block of text.

Processor Arguments: <source> <match_spec> --- <output_block>

The first line is a the source reference. <source> can be an external web page, or a page name in this wiki If it's an external web page, it should have an url of the form:

• name:password@http://domain/dir/file?params

If it's a page name, it can be optionally preceded by ! to read the page in raw mode. Otherwise, it is read as a regular page as HTML.

<match_spec> is list of one or more match statements. The simplest statement is a single pattern with one group. Each match statement consists of the variable/pattern declaration, and optional other parsing state machine control variables.

The variable/pattern declaration has the syntax:

• <field name>=<field pattern> where field name should be an alphnumeric string, and the field pattern should be a python regular expression, with a single group (portion of the expression in parenthesis). When the field pattern matches a line in the source content, the value of the part of the line that matches the group part of the pattern is used as the value for this field. (A field is also referred to as a 'match variable').

The field declaration can also declare an optional end-pattern, and field precursors. A field precursor specifies the name of another field that must be matched in order for this field to become matchable. The field end_pattern specifies when data collection for a field should end, and is used for multi-line data values.

<output_block> is a block of text that will be output as the result of substituting values from the fields into the text, where a python interpolation string (of the form %(var_name)s) has been specified.

<output_block> can be empty, in which case any fields found are printed with their names (if any) separated by spaces.

Example usage:: {{{#!WebSed !WebSedTestData black_thing=black (\S*) --- I found a black thing: %(black_thing)s on page WebSedTestData }}}

Example usage:: {{{#!WebSed https://stock-quotes.com/sony sony_stock_price=SNE=([0-9.]*) --- Today's Sony stock price (SNE): %(sony_stock_price)s from stock-quotes.com }}}

See https://bird.org/tbwikiDocWebSed for more details.

Back to page: SystemInfo