|
A "parsed table" is one that is read using the scan_data module,
(the same module used by the WebSed processor) to create a list of records.
|
A "parsed table" is one that is read using the ''scan_data'' module,
(the same module used by the WebSed processor) to create a list of records.
|
|
This is similar to a tbwiki table, where the table fields are read from level-1 sections and definition lists. But generic parsed tables are completely
free-form in their format. Due to this, there is no mechanism to write
the data back. A parsed table database is always read-only.
|
This is similar to a tbwiki table, where the table fields are read from level-1 sections and definition lists. But generic parsed tables are completely
free-form in their format. Due to this, there is no mechanism to write
the data back. A parsed table database is always read-only.
|
|
|
= source =
The source_spec for a parsed table can be wiki pages, absolute paths,
or external URL references. They may contain wildcards to read more than
one file. They can be lists of specifications (separated by colons).
|
|
Note that parsing data from external web pages (scraping them) may
introduce long delays in displaying a table.
|
Note that parsing data from external web pages (scraping them) may
introduce long delays in displaying a table.
|
|
DocWebSed for more information about the data_scan module and
how to declare regular expressions for finding data in a page of text.
|
= match_spec =
See [[DocWebSed]] for more information about the data_scan module and
how to declare regular expressions for finding data in a page of text.
|
|
Here are some miscelaneous rules for match_spec:
* if record_id is not a declared field for the parser, then the first field encountered is treated at the record_id for the database
* This means that this field is duplicated in the database. that is, if your first field is "year", your records will have a "year" field and a "record_id" field, with the same values for each field.
* Only one record for each record_id value is stored in the database. So, if your record_id is the year, and there are two records with the year "2010" in the parsed page, then only one of them appears in the parsed database.
* FIXTHIS - should add a error message when a record is overwritten in the database
* you can specify the field to use for the record_id, by passing a match expression of "record_id_name=<field_name>" in the match_spec.
* ex: record_id_name=full_date
* you must declare a record_start expression in order for the parser to determine when to reset itself and start looking for the next record
* more than one match expression can match a single line or a single piece of text
* For example: if the line were "date: August 20, 2015" you could use the following matches (simultaneously):
* full_date=date: (.*)
* month=date: (\w*) \d{1,2},
* day=date: \w* (\d{1,2}),
* year=date: \w* \d{1,2}, (\d\d\d\d}
* every match expression is evaluated against every line in the data text. this means that each expression must contain something sufficiently unique in the expression to prevent it from getting matches on incorrect lines. Overly general expressions will lead to confusion
|
Here are some miscelaneous rules for match_spec:
* if record_id is not a declared field for the parser, then the first field encountered is treated at the record_id for the database
* This means that this field is duplicated in the database. that is, if your first field is "year", your records will have a "year" field and a "record_id" field, with the same values for each field.
* Only one record for each record_id value is stored in the database. So, if your record_id is the year, and there are two records with the year "2010" in the parsed page, then only one of them appears in the parsed database.
* FIXTHIS - should add a error message when a record is overwritten in the database
* you can specify the field to use for the record_id, by passing a match expression of "record_id_name=<field_name>" in the match_spec.
* ex: record_id_name=full_date
* you must declare a record_start expression in order for the parser to determine when to reset itself and start looking for the next record
* more than one match expression can match a single line or a single piece of text
* For example: if the line were "date: August 20, 2015" you could use the following matches (simultaneously):
* full_date=date: (.*)
* month=date: (\w*) \d{1,2},
* day=date: \w* (\d{1,2}),
* year=date: \w* \d{1,2}, (\d\d\d\d}
* every match expression is evaluated against every line in the data text. this means that each expression must contain something sufficiently unique in the expression to prevent it from getting matches on incorrect lines. Overly general expressions will lead to confusion
|
|
|
= examples =
== Fixthis table ==
Here is a sample table declaration for the FIXTHIS list for the tbwiki software. It reads the code files (.py files) looking for FIXTHIS entries.
in the code itself.
|
{{{#!Table
source_spec=/home/tbird/work/tbwiki/cgi-bin/.*[.]py$:/home/tbird/work/tbwiki/cgi-bin/plugins/.*[.]py$
|
{{{
{{{#!Table
source_spec=/home/tbird/work/tbwiki/cgi-bin/.*[.]py$:/home/tbird/work/tbwiki/cgi-bin/plugins/.*[.]py$
|
|
match_spec="""
record_start=FIXTHIS
description=FIXTHIS[ -]*(.*)$
file=%(basename)s
line_no=%(line_no)s
"""
|
match_spec="""
record_start=FIXTHIS
description=FIXTHIS[ -]*(.*)$
file=%%(basename)s
line_no=%%(line_no)s
"""
|
|
cols=file:line_no:description
sortby=file:alpha,line_no:int
}} }
}}}
|
cols=file:line_no:description
sortby=file:alpha,line_no:int
}} }
}}}
|
|
|
== busybox page scrape ==
Here is an example of scraping the busybox mailing list archive summary
page to extract the size of the archive for each month (giving an approximation
of community activity level for that month).
|
{{{#!Table
source_spec=http://lists.busybox.net/pipermail/busybox/
match_spec="""
record_start=.*<tr>
year_month=href="(.*)[.]txt[.]gz"
year=href="(20\d\d)-.*[.]txt[.]gz"
month=href="20\d\d-(.*)[.]txt[.]gz"
size=Gzip'd Text (.*) KB
"""
cols=year_month:year:month:size
sortby=year:alpha,month:month
}}}
|
{{{
{{{#!Table
source_spec=http://lists.busybox.net/pipermail/busybox/
match_spec="""
record_start=.*<tr>
year_month=href="(.*)[.]txt[.]gz"
year=href="(20\d\d)-.*[.]txt[.]gz"
month=href="20\d\d-(.*)[.]txt[.]gz"
size=Gzip'd Text (.*) KB
"""
cols=year_month:year:month:size
sortby=year:alpha,month:month
}}}
|